report
daeyong.kim
2024-06-08
보고서의 목차는 다음과 같다.
1) 개요
- 데이터셋 정보
- 변수 정보
2) EDA
- 기술통계
- 수치변수의 정규성 검정
+ 통계량과 시각화 정보
3) Model Fitting
- Linear Regression
- Logistic Regression
- LDA
- QDA
- Multiple Regression
- Polynomial model
- natural spline model
- Tree라이브러리 및 데이터셋
library(ggplot2)
library(dplyr)##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, unionlibrary(rpart)
library(rpart.plot)
library(dlookr)## Registered S3 methods overwritten by 'dlookr':
## method from
## plot.transform scales
## print.transform scales##
## Attaching package: 'dlookr'## The following object is masked from 'package:base':
##
## transformlibrary(caret)## Loading required package: latticelibrary(tree)airline_test_set = read.csv('/Users/ivory/Documents/2024_1_1/static_final_report/dataset/Airline_Passneger_satisfaction/test.csv')
airline_train_set = read.csv('/Users/ivory/Documents/2024_1_1/static_final_report/dataset/Airline_Passneger_satisfaction/train.csv')
# X와 id 변수 제외
airline_test_set = subset(airline_test_set, select = -c(X, id))
airline_train_set = subset(airline_train_set, select = -c(X, id))개요
데이터셋 정보
해당 데이터셋은 kaggle에 업로드 되어 있는, 항공기 이용 승객들의
만족도 설문조사에 대한 것이다.
이 데이터셋을 이용해서 승객들의 만족도(“Satisfaction”) 예측을 수행할
예정이다.
training set과 test set이 사전에 분리되어 있었기 때문에, 따로 데이터의
분리를 진행하지 않고 제공하고 있는 그래도 사용하기로 한다.
출처 url : https://www.kaggle.com/datasets/teejmahal20/airline-passenger-satisfaction/data
변수정보
head(airline_train_set)## Gender Customer.Type Age Type.of.Travel Class Flight.Distance
## 1 Male Loyal Customer 13 Personal Travel Eco Plus 460
## 2 Male disloyal Customer 25 Business travel Business 235
## 3 Female Loyal Customer 26 Business travel Business 1142
## 4 Female Loyal Customer 25 Business travel Business 562
## 5 Male Loyal Customer 61 Business travel Business 214
## 6 Female Loyal Customer 26 Personal Travel Eco 1180
## Inflight.wifi.service Departure.Arrival.time.convenient
## 1 3 4
## 2 3 2
## 3 2 2
## 4 2 5
## 5 3 3
## 6 3 4
## Ease.of.Online.booking Gate.location Food.and.drink Online.boarding
## 1 3 1 5 3
## 2 3 3 1 3
## 3 2 2 5 5
## 4 5 5 2 2
## 5 3 3 4 5
## 6 2 1 1 2
## Seat.comfort Inflight.entertainment On.board.service Leg.room.service
## 1 5 5 4 3
## 2 1 1 1 5
## 3 5 5 4 3
## 4 2 2 2 5
## 5 5 3 3 4
## 6 1 1 3 4
## Baggage.handling Checkin.service Inflight.service Cleanliness
## 1 4 4 5 5
## 2 3 1 4 1
## 3 4 4 4 5
## 4 3 1 4 2
## 5 4 3 3 3
## 6 4 4 4 1
## Departure.Delay.in.Minutes Arrival.Delay.in.Minutes satisfaction
## 1 25 18 neutral or dissatisfied
## 2 1 6 neutral or dissatisfied
## 3 0 0 satisfied
## 4 11 9 neutral or dissatisfied
## 5 0 0 satisfied
## 6 0 0 neutral or dissatisfiedstr(airline_train_set)## 'data.frame': 103904 obs. of 23 variables:
## $ Gender : chr "Male" "Male" "Female" "Female" ...
## $ Customer.Type : chr "Loyal Customer" "disloyal Customer" "Loyal Customer" "Loyal Customer" ...
## $ Age : int 13 25 26 25 61 26 47 52 41 20 ...
## $ Type.of.Travel : chr "Personal Travel" "Business travel" "Business travel" "Business travel" ...
## $ Class : chr "Eco Plus" "Business" "Business" "Business" ...
## $ Flight.Distance : int 460 235 1142 562 214 1180 1276 2035 853 1061 ...
## $ Inflight.wifi.service : int 3 3 2 2 3 3 2 4 1 3 ...
## $ Departure.Arrival.time.convenient: int 4 2 2 5 3 4 4 3 2 3 ...
## $ Ease.of.Online.booking : int 3 3 2 5 3 2 2 4 2 3 ...
## $ Gate.location : int 1 3 2 5 3 1 3 4 2 4 ...
## $ Food.and.drink : int 5 1 5 2 4 1 2 5 4 2 ...
## $ Online.boarding : int 3 3 5 2 5 2 2 5 3 3 ...
## $ Seat.comfort : int 5 1 5 2 5 1 2 5 3 3 ...
## $ Inflight.entertainment : int 5 1 5 2 3 1 2 5 1 2 ...
## $ On.board.service : int 4 1 4 2 3 3 3 5 1 2 ...
## $ Leg.room.service : int 3 5 3 5 4 4 3 5 2 3 ...
## $ Baggage.handling : int 4 3 4 3 4 4 4 5 1 4 ...
## $ Checkin.service : int 4 1 4 1 3 4 3 4 4 4 ...
## $ Inflight.service : int 5 4 4 4 3 4 5 5 1 3 ...
## $ Cleanliness : int 5 1 5 2 3 1 2 4 2 2 ...
## $ Departure.Delay.in.Minutes : int 25 1 0 11 0 0 9 4 0 0 ...
## $ Arrival.Delay.in.Minutes : num 18 6 0 9 0 0 23 0 0 0 ...
## $ satisfaction : chr "neutral or dissatisfied" "neutral or dissatisfied" "satisfied" "neutral or dissatisfied" ...observations of training set : 103904
number of variables : 23
- “Gender” : 승객의 성별 (Female, Male)
- “Customer.Type” : 고객 충성도 유형 (Loyal Customer, disLoyal
Customer)
- “Age” : 승객의 나이
- “Type.of.Travel” : 승객의 비행 목적 (Business travel, Personal Travel)
- “Class” : 승객의 비행기 좌석 등급 (Business, Eco, Ohter)
- “Flight.Distance” : 승객별 비행 거리
- “Inflight.wifi.service” : 기내 와이파이 서비스 만족도 (0: 해당 없음;
1-5)
- “Departure.Arrival.time.convenient” : 출발/도착 시간 편의성
만족도
- “Ease.of.Online.booking” : 온라인 예약 용이성 만족도(0: 해당 없음;
1-5)
- “Gate.location” : 게이트 위치 만족도(0: 해당 없음; 1-5)
- “Food.and.drink” : 음식 및 음료 만족도(0: 해당 없음; 1-5)
- “Online.boarding” : 온라인 탑승 수속 만족도(0: 해당 없음; 1-5)
- “Seat.comfort” : 좌석 편안함 만족도(0: 해당 없음; 1-5)
- “Inflight.entertainment” : 기내 오락 만족도(0: 해당 없음; 1-5)
- “On.board.service” : 기내 서비스 만족도(0: 해당 없음; 1-5)
- “Leg.room.service” : 다리 공간 서비스 만족도(0: 해당 없음; 1-5)
- “Baggage.handling” : 수하물 처리 만족도(0: 해당 없음; 1-5)
- “Checkin.service” : 체크인 서비스 만족도(0: 해당 없음; 1-5)
- “Inflight.service” : 기내 서비스 만족도(0: 해당 없음; 1-5)
- “Cleanliness” : 청결도 만족도(0: 해당 없음; 1-5)
- “Departure.Delay.in.Minutes” : 출발 지연 시간 (분 단위)
- “Arrival.Delay.in.Minutes” : 도착 지연 시간 (분 단위)
- “satisfaction” : 항공사 만족도 (만족, 보통 또는 불만족)
X와 id 변수는 satisfaction을 예측하는데 유의미하지 않으므로 제외한다.
numerical 변수 : [“Age”, “Flight.Distance”, “Departure.Delay.in.Minutes”, “Arrival.Delay.in.Minutes”] categorical 변수 : [“Gender”, “Customer.Type”, “Type.of.Travel”, “Class”, “Inflight.wifi.service”, “Departure.Arrival.time.convenient”, “Ease.of.Online.booking”, “Gate.location”, “Food.and.drink”, “Online.boarding”, “Seat.comfort”, “Inflight.entertainment”, “On.board.service”, “Leg.room.service”, “Baggage.handing”, “Checkin.service”, “Inflight.service”, “Cleanliness”, “satisfaction”]
unique_values = unique(airline_train_set$satisfaction)
unique_values## [1] "neutral or dissatisfied" "satisfied"satisfaction는 “satisfied” 또는 “neutral or dissatisfied”가 존재한다. 해당 명목형 변수를 이진 변수로 변환한다.
airline_test_set$satisfaction = ifelse(airline_test_set$satisfaction == "satisfied", 1, 0)
airline_train_set$satisfaction = ifelse(airline_train_set$satisfaction == "satisfied", 1, 0)table(airline_train_set$satisfaction)##
## 0 1
## 58879 45025
total : 103904 satisfied : 45025
neutral or dissatisfied : 58879
“satisfied” : “neutral or dissatisfied” = 4.3 : 5.7
table(airline_test_set$satisfaction)##
## 0 1
## 14573 11403
total : 25976 satisfied : 11403
neutral or dissatisfied : 14573
“satisfied” : “neutral or dissatisfied” = 4.4 : 5.6
conclustion
training set 과 test set의 비율은 8 대 2이다. 각 data set에서 satisfaction 변수의 클래스 비율은 약 4.4 대 5.6정도로 큰 차이가 나지 않는다.
EDA
summary(airline_train_set)## Gender Customer.Type Age Type.of.Travel
## Length:103904 Length:103904 Min. : 7.00 Length:103904
## Class :character Class :character 1st Qu.:27.00 Class :character
## Mode :character Mode :character Median :40.00 Mode :character
## Mean :39.38
## 3rd Qu.:51.00
## Max. :85.00
##
## Class Flight.Distance Inflight.wifi.service
## Length:103904 Min. : 31 Min. :0.00
## Class :character 1st Qu.: 414 1st Qu.:2.00
## Mode :character Median : 843 Median :3.00
## Mean :1189 Mean :2.73
## 3rd Qu.:1743 3rd Qu.:4.00
## Max. :4983 Max. :5.00
##
## Departure.Arrival.time.convenient Ease.of.Online.booking Gate.location
## Min. :0.00 Min. :0.000 Min. :0.000
## 1st Qu.:2.00 1st Qu.:2.000 1st Qu.:2.000
## Median :3.00 Median :3.000 Median :3.000
## Mean :3.06 Mean :2.757 Mean :2.977
## 3rd Qu.:4.00 3rd Qu.:4.000 3rd Qu.:4.000
## Max. :5.00 Max. :5.000 Max. :5.000
##
## Food.and.drink Online.boarding Seat.comfort Inflight.entertainment
## Min. :0.000 Min. :0.00 Min. :0.000 Min. :0.000
## 1st Qu.:2.000 1st Qu.:2.00 1st Qu.:2.000 1st Qu.:2.000
## Median :3.000 Median :3.00 Median :4.000 Median :4.000
## Mean :3.202 Mean :3.25 Mean :3.439 Mean :3.358
## 3rd Qu.:4.000 3rd Qu.:4.00 3rd Qu.:5.000 3rd Qu.:4.000
## Max. :5.000 Max. :5.00 Max. :5.000 Max. :5.000
##
## On.board.service Leg.room.service Baggage.handling Checkin.service
## Min. :0.000 Min. :0.000 Min. :1.000 Min. :0.000
## 1st Qu.:2.000 1st Qu.:2.000 1st Qu.:3.000 1st Qu.:3.000
## Median :4.000 Median :4.000 Median :4.000 Median :3.000
## Mean :3.382 Mean :3.351 Mean :3.632 Mean :3.304
## 3rd Qu.:4.000 3rd Qu.:4.000 3rd Qu.:5.000 3rd Qu.:4.000
## Max. :5.000 Max. :5.000 Max. :5.000 Max. :5.000
##
## Inflight.service Cleanliness Departure.Delay.in.Minutes
## Min. :0.00 Min. :0.000 Min. : 0.00
## 1st Qu.:3.00 1st Qu.:2.000 1st Qu.: 0.00
## Median :4.00 Median :3.000 Median : 0.00
## Mean :3.64 Mean :3.286 Mean : 14.82
## 3rd Qu.:5.00 3rd Qu.:4.000 3rd Qu.: 12.00
## Max. :5.00 Max. :5.000 Max. :1592.00
##
## Arrival.Delay.in.Minutes satisfaction
## Min. : 0.00 Min. :0.0000
## 1st Qu.: 0.00 1st Qu.:0.0000
## Median : 0.00 Median :0.0000
## Mean : 15.18 Mean :0.4333
## 3rd Qu.: 13.00 3rd Qu.:1.0000
## Max. :1584.00 Max. :1.0000
## NA's :310[Gender, Customer.Type, Type.of.Travel, satisfaction] 변수를 제외한 나머지는 모두 수치변수에 해당한다.
“Satisfaction”을 반응변수로 선택한다.
#training set에 대한 결측치 확인
sum(is.na(airline_train_set))## [1] 310colSums(is.na(airline_train_set))## Gender Customer.Type
## 0 0
## Age Type.of.Travel
## 0 0
## Class Flight.Distance
## 0 0
## Inflight.wifi.service Departure.Arrival.time.convenient
## 0 0
## Ease.of.Online.booking Gate.location
## 0 0
## Food.and.drink Online.boarding
## 0 0
## Seat.comfort Inflight.entertainment
## 0 0
## On.board.service Leg.room.service
## 0 0
## Baggage.handling Checkin.service
## 0 0
## Inflight.service Cleanliness
## 0 0
## Departure.Delay.in.Minutes Arrival.Delay.in.Minutes
## 0 310
## satisfaction
## 0training set의 Arrival.Delay.in.Minutes 변수에 310개의 결측변수가 존재함을 확인했다. 해당 결측치를 제거하기로 한다.
#test set에 대한 결측치 확인
sum(is.na(airline_test_set))## [1] 83colSums(is.na(airline_test_set))## Gender Customer.Type
## 0 0
## Age Type.of.Travel
## 0 0
## Class Flight.Distance
## 0 0
## Inflight.wifi.service Departure.Arrival.time.convenient
## 0 0
## Ease.of.Online.booking Gate.location
## 0 0
## Food.and.drink Online.boarding
## 0 0
## Seat.comfort Inflight.entertainment
## 0 0
## On.board.service Leg.room.service
## 0 0
## Baggage.handling Checkin.service
## 0 0
## Inflight.service Cleanliness
## 0 0
## Departure.Delay.in.Minutes Arrival.Delay.in.Minutes
## 0 83
## satisfaction
## 0Arrival.Delay.in.Minutes에 83개의 결측변수가 존재함을 확인했다. 해당 결측치를 제거하기로 한다.
#training set의 결측치 제거
airline_train_set = na.omit(airline_train_set)#test set의 결측치 제거
airline_test_set = na.omit(airline_test_set)결측치 제거 확인
#training set에 대한 결측치 확인
sum(is.na(airline_train_set))## [1] 0colSums(is.na(airline_train_set))## Gender Customer.Type
## 0 0
## Age Type.of.Travel
## 0 0
## Class Flight.Distance
## 0 0
## Inflight.wifi.service Departure.Arrival.time.convenient
## 0 0
## Ease.of.Online.booking Gate.location
## 0 0
## Food.and.drink Online.boarding
## 0 0
## Seat.comfort Inflight.entertainment
## 0 0
## On.board.service Leg.room.service
## 0 0
## Baggage.handling Checkin.service
## 0 0
## Inflight.service Cleanliness
## 0 0
## Departure.Delay.in.Minutes Arrival.Delay.in.Minutes
## 0 0
## satisfaction
## 0#test set에 대한 결측치 확인
sum(is.na(airline_test_set))## [1] 0colSums(is.na(airline_test_set))## Gender Customer.Type
## 0 0
## Age Type.of.Travel
## 0 0
## Class Flight.Distance
## 0 0
## Inflight.wifi.service Departure.Arrival.time.convenient
## 0 0
## Ease.of.Online.booking Gate.location
## 0 0
## Food.and.drink Online.boarding
## 0 0
## Seat.comfort Inflight.entertainment
## 0 0
## On.board.service Leg.room.service
## 0 0
## Baggage.handling Checkin.service
## 0 0
## Inflight.service Cleanliness
## 0 0
## Departure.Delay.in.Minutes Arrival.Delay.in.Minutes
## 0 0
## satisfaction
## 0plot_outlier(airline_train_set)
[“Age”, “Inflight.wifi.service”, “Online.boarding”, “Gate.location”]
[“Flight.Distance”]
[“Online.boarding”, “Seat.comfort”, “Inflight.entertainment”,
“Baggage.handling”, “Cleanliness”, “On.board.service”,
“Leg.room.fservice”, “Inflight.service”]
[“Flight.Distance”, “Departure.Delay.in.Minutes”,
“Arrival.Delay.in.Minutes”]
위의 변수들 중에서 [“Age”, “Inflight.wifi.service”, “Online.boarding”, “Gate.location”] 4가지의 변수만 정규분포를 따르고 있었다.
이 중에서 Age만 numerical 변수이고 나머지는 categorical 변수이다.
plot_normality(airline_train_set)
Aae의 경우, 완벽한 정규분포라고는 할 수 없지만, Q-Q plot을 통해서 어느정도 정규성을 따른다는 것을 확인했다.
변수들 중에서 [“Flight.Distance”, “Departure.Delay.in.Minutes”, “Arrival.Delay.in.Minutes”]는 log transformation 한 경우에 정규 분포의 형태를 띄므로 해당 변수들에 대해서 log transformation을 진행하도록 한다.
airline_test_set$Flight.Distance_log = log(airline_test_set$Flight.Distance + 1)
airline_test_set$Departure.Delay.in.Minutes_log = log(airline_test_set$Departure.Delay.in.Minutes + 1)
airline_test_set$Arrival.Delay.in.Minutes_log = log(airline_test_set$Arrival.Delay.in.Minutes + 1)airline_train_set$Flight.Distance_log = log(airline_train_set$Flight.Distance + 1)
airline_train_set$Departure.Delay.in.Minutes_log = log(airline_train_set$Departure.Delay.in.Minutes + 1)
airline_train_set$Arrival.Delay.in.Minutes_log = log(airline_train_set$Arrival.Delay.in.Minutes + 1)Model Fitting
Linear Regression model
원래의 목적은 satisfaction 변수를 추정하는 것이지만, 해당 변수는 regression모델에는 적합하지 않다. 따라서, 기내 서비스 몬족도를 예측하는(Inflight.service) 모델을 만드는 것을 목표로 한다.
#log scale로 변환하기 전 변수들 제외
inflight_service_lm = lm(Inflight.service ~ . -Flight.Distance -Departure.Delay.in.Minutes -Arrival.Delay.in.Minutes, data = airline_train_set)summary(inflight_service_lm)##
## Call:
## lm(formula = Inflight.service ~ . - Flight.Distance - Departure.Delay.in.Minutes -
## Arrival.Delay.in.Minutes, data = airline_train_set)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3.9642 -0.3341 0.0092 0.4125 3.4174
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.408e+00 2.744e-02 51.322 < 2e-16 ***
## GenderMale 4.548e-02 5.097e-03 8.921 < 2e-16 ***
## Customer.TypeLoyal Customer -2.409e-01 8.480e-03 -28.409 < 2e-16 ***
## Age -3.407e-03 1.807e-04 -18.851 < 2e-16 ***
## Type.of.TravelPersonal Travel 2.352e-01 8.396e-03 28.011 < 2e-16 ***
## ClassEco -1.435e-01 7.435e-03 -19.303 < 2e-16 ***
## ClassEco Plus -1.170e-01 1.118e-02 -10.469 < 2e-16 ***
## Inflight.wifi.service -1.477e-02 3.040e-03 -4.860 1.18e-06 ***
## Departure.Arrival.time.convenient 3.215e-02 2.148e-03 14.967 < 2e-16 ***
## Ease.of.Online.booking -6.975e-03 2.999e-03 -2.325 0.0201 *
## Gate.location 1.004e-03 2.432e-03 0.413 0.6798
## Food.and.drink -8.296e-02 2.799e-03 -29.645 < 2e-16 ***
## Online.boarding -6.166e-04 2.728e-03 -0.226 0.8212
## Seat.comfort -7.032e-02 2.962e-03 -23.740 < 2e-16 ***
## Inflight.entertainment 3.126e-01 3.625e-03 86.222 < 2e-16 ***
## On.board.service 1.754e-01 2.576e-03 68.104 < 2e-16 ***
## Leg.room.service 6.528e-02 2.214e-03 29.484 < 2e-16 ***
## Baggage.handling 3.461e-01 2.765e-03 125.198 < 2e-16 ***
## Checkin.service 7.510e-02 2.225e-03 33.760 < 2e-16 ***
## Cleanliness -1.089e-01 3.262e-03 -33.383 < 2e-16 ***
## satisfaction 8.742e-02 7.669e-03 11.399 < 2e-16 ***
## Flight.Distance_log -2.662e-03 3.106e-03 -0.857 0.3914
## Departure.Delay.in.Minutes_log 3.679e-05 2.666e-03 0.014 0.9890
## Arrival.Delay.in.Minutes_log -1.626e-02 2.655e-03 -6.125 9.09e-10 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.816 on 103570 degrees of freedom
## Multiple R-squared: 0.5184, Adjusted R-squared: 0.5182
## F-statistic: 4846 on 23 and 103570 DF, p-value: < 2.2e-16모든 변수를 이용해서 모델을 생성한 결과,
[“Gate.location”, “Online.boarding”, “Flight.Distance_log”,
“Departure.Delay.in.Minutes_log”]는 Inflight.service 변수 예측에
통계적으로 유의미하지 않음을 확인했다.
따라서, 해당 변수를 제외하고 새로운 모델을 만들기로 한다.
#log scale로 변환하기 전 변수들 제외
inflight_service_lm = lm(Inflight.service ~ . -Gate.location -Online.boarding -Flight.Distance_log -Departure.Delay.in.Minutes_log -Flight.Distance -Departure.Delay.in.Minutes -Arrival.Delay.in.Minutes, data = airline_train_set)summary(inflight_service_lm)##
## Call:
## lm(formula = Inflight.service ~ . - Gate.location - Online.boarding -
## Flight.Distance_log - Departure.Delay.in.Minutes_log - Flight.Distance -
## Departure.Delay.in.Minutes - Arrival.Delay.in.Minutes, data = airline_train_set)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3.9650 -0.3341 0.0090 0.4125 3.4214
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.392267 0.017337 80.309 < 2e-16 ***
## GenderMale 0.045562 0.005086 8.958 < 2e-16 ***
## Customer.TypeLoyal Customer -0.242568 0.008304 -29.211 < 2e-16 ***
## Age -0.003404 0.000179 -19.024 < 2e-16 ***
## Type.of.TravelPersonal Travel 0.235432 0.008353 28.187 < 2e-16 ***
## ClassEco -0.141809 0.007183 -19.743 < 2e-16 ***
## ClassEco Plus -0.115070 0.010952 -10.506 < 2e-16 ***
## Inflight.wifi.service -0.014794 0.002949 -5.017 5.25e-07 ***
## Departure.Arrival.time.convenient 0.032506 0.002025 16.054 < 2e-16 ***
## Ease.of.Online.booking -0.006954 0.002830 -2.457 0.014 *
## Food.and.drink -0.082972 0.002797 -29.668 < 2e-16 ***
## Seat.comfort -0.070524 0.002882 -24.473 < 2e-16 ***
## Inflight.entertainment 0.312758 0.003604 86.783 < 2e-16 ***
## On.board.service 0.175355 0.002572 68.178 < 2e-16 ***
## Leg.room.service 0.065230 0.002205 29.587 < 2e-16 ***
## Baggage.handling 0.346136 0.002765 125.207 < 2e-16 ***
## Checkin.service 0.074993 0.002214 33.879 < 2e-16 ***
## Cleanliness -0.109026 0.003243 -33.625 < 2e-16 ***
## satisfaction 0.086893 0.007471 11.631 < 2e-16 ***
## Arrival.Delay.in.Minutes_log -0.016246 0.001567 -10.365 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.816 on 103574 degrees of freedom
## Multiple R-squared: 0.5183, Adjusted R-squared: 0.5183
## F-statistic: 5867 on 19 and 103574 DF, p-value: < 2.2e-16해당 모델의 설명력(R-squared)는 0.5183임을 확인할 수 있었다.
correlation
library(corrplot)## corrplot 0.92 loadedselected_data = select(airline_train_set, Inflight.service, Age, Inflight.wifi.service, Departure.Arrival.time.convenient, Ease.of.Online.booking, Food.and.drink, Seat.comfort, Inflight.entertainment, On.board.service, Leg.room.service, Baggage.handling, Checkin.service, Cleanliness)
selected_data = rename(selected_data, Inflight = Inflight.service,
wifi = Inflight.wifi.service,
depart.a = Departure.Arrival.time.convenient,
book = Ease.of.Online.booking,
food = Food.and.drink,
seat = Seat.comfort,
enter = Inflight.entertainment,
board = On.board.service,
leg = Leg.room.service,
bagg = Baggage.handling,
checkin = Checkin.service,
clean = Cleanliness)correlation_matrix = cor(selected_data, use="complete.obs")
corrplot(correlation_matrix, method = "number")
Inflight.service와 각각의 예측변수 사이의 상관관계 확인 후, 0.4
이상의 값을 갖는 변수들만을 선택해서 linear regression 모델의 예측변수로
선정하기로 한다.
이때, 선택된 예측변수는 [“Inflight.entertainment”, “On.board.service”,
“Baggage.handling”] 이다.
inflight_service_final_lm = lm(Inflight.service ~ Inflight.entertainment + On.board.service + Baggage.handling, data = airline_train_set)
summary(inflight_service_final_lm)##
## Call:
## lm(formula = Inflight.service ~ Inflight.entertainment + On.board.service +
## Baggage.handling, data = airline_train_set)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3.8204 -0.4340 -0.0249 0.4152 3.3615
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 0.842966 0.009628 87.55 <2e-16 ***
## Inflight.entertainment 0.110287 0.002233 49.40 <2e-16 ***
## On.board.service 0.245065 0.002502 97.94 <2e-16 ***
## Baggage.handling 0.440141 0.002675 164.51 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.8504 on 103590 degrees of freedom
## Multiple R-squared: 0.4767, Adjusted R-squared: 0.4767
## F-statistic: 3.145e+04 on 3 and 103590 DF, p-value: < 2.2e-16상관관계를 기반으로 예측변수를 선정했지만, 오히려 R-squared 값이
하락했다.
R-squared 변화 : 0.5183 -> 0.4767(-0.0416)
설명력이 약간 하락했지만, 그 차이가 미미하고 사용한 예측변수의 수도 적기
때문에 해당 모델을 최종 모델로 선정한다.
prediction
predictions = predict(inflight_service_final_lm, newdata = airline_test_set)predictions[1:5]## 1 2 3 4 5
## 4.820431 4.024938 3.364223 1.638459 2.433952소수 첫쨰 자리에서 반올림한다.
predicted = round(predictions, 0)
actual = airline_test_set$Inflight.servicelength(predicted)## [1] 25893length(actual)## [1] 25893Evaluation
predicted = factor(predicted)
actual = factor(actual)
conf_matrix = confusionMatrix(predicted, actual)## Warning in levels(reference) != levels(data): longer object length is not a
## multiple of shorter object length## Warning in confusionMatrix.default(predicted, actual): Levels are not in the
## same order for reference and data. Refactoring data to match.print(conf_matrix)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1 2 3 4 5
## 0 0 0 0 0 0 0
## 1 0 0 0 0 0 0
## 2 2 915 1612 266 144 140
## 3 0 495 827 3586 1977 545
## 4 0 344 359 1092 6803 2917
## 5 0 21 40 61 428 3319
##
## Overall Statistics
##
## Accuracy : 0.5917
## 95% CI : (0.5856, 0.5977)
## No Information Rate : 0.3612
## P-Value [Acc > NIR] : < 2.2e-16
##
## Kappa : 0.4414
##
## Mcnemar's Test P-Value : NA
##
## Statistics by Class:
##
## Class: 0 Class: 1 Class: 2 Class: 3 Class: 4 Class: 5
## Sensitivity 0.000e+00 0.00000 0.56801 0.7165 0.7274 0.4796
## Specificity 1.000e+00 1.00000 0.93637 0.8160 0.7151 0.9710
## Pos Pred Value NaN NaN 0.52355 0.4826 0.5908 0.8578
## Neg Pred Value 9.999e-01 0.93145 0.94626 0.9231 0.8227 0.8365
## Prevalence 7.724e-05 0.06855 0.10960 0.1933 0.3612 0.2673
## Detection Rate 0.000e+00 0.00000 0.06226 0.1385 0.2627 0.1282
## Detection Prevalence 0.000e+00 0.00000 0.11891 0.2870 0.4447 0.1494
## Balanced Accuracy 5.000e-01 0.50000 0.75219 0.7662 0.7213 0.7253Accuracy : 0.59
Class 0
민감도가 0으로 클래스가 0인 경우에 하나도 올바르게 감지하지 못했다는
것을 의미한다.
특이도는 1으로 크래스가 0이 아닌 경우에 완벽하게 감지했다는 것을
의미한다.
이러한 결과가 나온 이유는 class가 0인 레이블이 거의 존재하지 않기
때문이다.
Class 1
민감도는 0, 특이도는 1으로 class가 1인 경우 1로 제대로 예측을 하지 못 했다는 것을 확인할 수 있다.
Class 2
민감도가 약 0.57으로 클래스 2 예측 성능이 상대적으로 나은 것을 알 수
있다.
특이도도는 약 0.94으로 클래스 2가 아닌 것을 잘 구분하는 것을 확인할 수
있다.
balanced accuacy는 0.75으로 이 클래스에 대한 모델의 성능이 상대적으로
괜찮은 것을 알 수 있다.
Class 3
민감도가 0.72로 클래스의 예측 능력이 이전의 클래스보다 높은 것을 알 수 있다. 특이도는 0.82이다.
Class 4
민감도가 0.73으로 해당 모델이 class 4를 예측하는 데 가장 높은 성능을 보인다.
Class 5
민감도는 0.48, 특이도는0.97이다.
conclusion : Linear regression
해당 모델은 클래스(0,1,2)에서 낮은 성능을 보이지만, 다른 클래스(3,4,5)에서는 낭느 결과를 보이고 있는 것을 확인했다. 해당 결과를 보인 이유는, Inflight.service의 분포가 left-skewed distribution을 따르기 때문이라고 결론을 내린다.
Logistic regresion
해당 파트에서는 Logistic model을 이용하여 항공기 이용객들의 satisfaction을 예측하는 것을 목표로 한다.
일반적인 상식으로 생각해본다면, satisfaction과 가장 관련이 큰 요소들은 [Customer.Type, Departure.Arrival.time.convenient, On.board.service, Seat.comfort, Cleanliness] 이다. 해당 변수들을 중점적으로 종속변수를 예측할 것이다.
table(airline_train_set$satisfaction)##
## 0 1
## 58697 44897<obs : 103594>
“보통 또는 불만족” : 58697(0.57) “만족” : 44897(0.43)
만족으로 답한 승객보다 보통 또는 불만족이라고 답한 승객의 수가 많은 것을 확인했다.
satisfcation ~ Customer.Type
CrossTable = table(airline_train_set$satisfaction, airline_train_set$Customer.Type)
mosaicplot(CrossTable, main="Mosaic Plot")
만족이라고 답한 승객 중 대부분은 충성도가 높은 고객이었다.
하지만, 보통 또는 불만족인 경우에도 충성도가 높은 고객의 비율이 높은
것을 확인할 수 있다.
ggplot(airline_train_set, aes(x=factor(Customer.Type), fill=factor(satisfaction))) +
geom_bar(position="fill") +
labs(x="Category Variable", y="Proportion", fill="Target")
예측변수가 특정 클래스로 편향되어 있기 때문에, 비율을 기준으로 확인했다.
해당 항공사 데이터의 경우, 충성 고객임에도 만족하는 경우가 50% 미만인
것을 알 수 있다. 충성고객이어야 그나마 만족이라고 답변을 한다는 것을 알
수 있다.
satisfcation ~ Departure.Arrival.time.convenient
ggplot(airline_train_set, aes(x=factor(Departure.Arrival.time.convenient), fill=factor(satisfaction))) +
geom_bar(position="fill") +
labs(x="time.convenient", y="Proportion", title="Proportion")
satisfaction을 예측하는 데, time.convenuent의 클래스별 유의미한 의미가 보이지 않는다. 해당 변수를 제외하기로 한다.
satisfcation ~ On.board.service
ggplot(airline_train_set, aes(x=factor(On.board.service), fill=factor(satisfaction))) +
geom_bar(position="fill") +
labs(x="On.board.service", y="Proportion", title="Proportion")
On.baord.service(기내 서비스 만족도)가 높을수록 satisfaction에
만족이라고 답한 비율이 올라가는 것을 확인할 수 있다. 이것으로
On.board.service와 satisfaction은 유의미한 관계를 갖다는 것을 알 수
있다.
satisfcation ~ Seat.comfort
ggplot(airline_train_set, aes(x=factor(Seat.comfort), fill=factor(satisfaction))) +
geom_bar(position="fill") +
labs(x="Seat.comfort", y="Proportion", title="Proportion")
Seat.comfort의 평가가 높아질수록 satisfaction의 비율이 높아지는 것을 확인할 수 있다.
satisfcation ~ Cleanliness
ggplot(airline_train_set, aes(x=factor(Cleanliness), fill=factor(satisfaction))) +
geom_bar(position="fill") +
labs(x="Cleanliness", y="Proportion", title="Proportion")
satisifaction와 Cleanliness 변수의 관계역시 양적 상관관계를 갖는 것을 확인할 수 있다.
결론적으로, satisfaction을 예측하는 예측변수로 [“Customer.Type”, “On.board.service”, “Seat.comfort”, “Cleanliness”]를 사용하기로 한다.
model fitting : logistic regression
logistic_model = glm(satisfaction ~ Customer.Type + On.board.service + Seat.comfort + Cleanliness, data = airline_train_set, family = "binomial")summary(logistic_model)##
## Call:
## glm(formula = satisfaction ~ Customer.Type + On.board.service +
## Seat.comfort + Cleanliness, family = "binomial", data = airline_train_set)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -5.092707 0.037064 -137.40 <2e-16 ***
## Customer.TypeLoyal Customer 0.919842 0.020282 45.35 <2e-16 ***
## On.board.service 0.532996 0.005968 89.30 <2e-16 ***
## Seat.comfort 0.408277 0.007442 54.86 <2e-16 ***
## Cleanliness 0.229151 0.007316 31.32 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 141768 on 103593 degrees of freedom
## Residual deviance: 115889 on 103589 degrees of freedom
## AIC: 115899
##
## Number of Fisher Scoring iterations: 4모든 예측변수들이 통계적으로 유의미함을 보인다.
Evaluation
probabilities = predict(logistic_model, newdata = airline_test_set, type = "response")
predicted_classes <- ifelse(probabilities > 0.5, 1, 0)
conf_matrix = confusionMatrix(factor(predicted_classes), factor(airline_test_set$satisfaction))
print(conf_matrix)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 11544 3926
## 1 2984 7439
##
## Accuracy : 0.7331
## 95% CI : (0.7277, 0.7385)
## No Information Rate : 0.5611
## P-Value [Acc > NIR] : < 2.2e-16
##
## Kappa : 0.4532
##
## Mcnemar's Test P-Value : < 2.2e-16
##
## Sensitivity : 0.7946
## Specificity : 0.6546
## Pos Pred Value : 0.7462
## Neg Pred Value : 0.7137
## Prevalence : 0.5611
## Detection Rate : 0.4458
## Detection Prevalence : 0.5975
## Balanced Accuracy : 0.7246
##
## 'Positive' Class : 0
## tp = 7439
fp = 2984
fn = 3926
tn = 11544
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * (precision*recall / (precision+recall))
cat("Precision:", precision, "\n")## Precision: 0.7137101cat("Recall:", recall, "\n")## Recall: 0.6545535cat("F1 Score:", f1, "\n")## F1 Score: 0.6828529f1-score가 0.68으로, 해당 로지스틱 모델의 성능이 좋게 나타나진 않았다.
LDA
LDA는 서로 다른 카테고리 사이의 거리를 최대화하는 차원축소의 기법이다. 보통은 분류 카테고리가 3개 이상일 때 사용하고, 각각의 클래스가 정규분포를 따른다고 가정을 하는데 해당 데이터셋에는 적합하지가 않다. LDA의 가정을 만족하지 않으므로 대신 QDA를 진행하도록 한다.
QDA
library(MASS)##
## Attaching package: 'MASS'## The following object is masked from 'package:dplyr':
##
## selectqda_model = qda(satisfaction ~ Customer.Type + On.board.service + Seat.comfort + Cleanliness, data = airline_train_set)
summary(qda_model)## Length Class Mode
## prior 2 -none- numeric
## counts 2 -none- numeric
## means 8 -none- numeric
## scaling 32 -none- numeric
## ldet 2 -none- numeric
## lev 2 -none- character
## N 1 -none- numeric
## call 3 -none- call
## terms 3 terms call
## xlevels 1 -none- list# 예측 및 모델 평가
probabilities_qda = predict(qda_model, newdata = airline_test_set)
confusionMatrix(probabilities_qda$class, factor(airline_test_set$satisfaction))## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 10452 3137
## 1 4076 8228
##
## Accuracy : 0.7214
## 95% CI : (0.7159, 0.7269)
## No Information Rate : 0.5611
## P-Value [Acc > NIR] : < 2.2e-16
##
## Kappa : 0.4395
##
## Mcnemar's Test P-Value : < 2.2e-16
##
## Sensitivity : 0.7194
## Specificity : 0.7240
## Pos Pred Value : 0.7692
## Neg Pred Value : 0.6687
## Prevalence : 0.5611
## Detection Rate : 0.4037
## Detection Prevalence : 0.5248
## Balanced Accuracy : 0.7217
##
## 'Positive' Class : 0
## tp = 8228
fp = 4076
fn = 3137
tn = 10452
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * (precision*recall / (precision+recall))
cat("Precision:", precision, "\n")## Precision: 0.6687256cat("Recall:", recall, "\n")## Recall: 0.7239771cat("F1 Score:", f1, "\n")## F1 Score: 0.6952554logistic과 QDA를 비교했을 때, QDA의 경우가 precision이 낮아지고, recall이 높아져서 근소하게 f1-score값이 logistic model 보다는 높게 나타났다. 근러나 그 차이가 근소하기 때문에 유의미한 차이가 있다고는 말하기 어렵다.
multiple regression
satisfaction model에는 수치형 변수를 종속 변수로 설정하는 것이
이상적이지만, 해당 데이터셋의 대부분의 변수의 종류가 categorical 이므로,
그 중에서 Ordinal 변수를 예측하는 것으로 한다.
그 중에서 Inflight.service를 예측하는 것을 목표로 한다.
EDA
우선, Inflight.service와 다른 변수들 간의 상관관계를 확인한다.
library(dplyr)
# 1. correlation
num_vars <- airline_train_set %>% dplyr::select(Inflight.service, Customer.Type, Type.of.Travel, Class, Flight.Distance_log, Inflight.wifi.service, Seat.comfort, Inflight.entertainment, On.board.service, Cleanliness)#one-hot encoding
## Customer.Type
Customer.type_factor = factor(num_vars$Customer.Type, levels = unique(num_vars$Customer.Type))
num_vars$Customer.Type = as.integer(Customer.type_factor)
## Type.of.Travel
Type.of.travel_factor = factor(num_vars$Type.of.Travel, levels = unique(num_vars$Type.of.Travel))
num_vars$Type.of.Travel = as.integer(Type.of.travel_factor)
## Class
Class_factor = factor(num_vars$Class, levels = unique(num_vars$Class))
num_vars$Class = as.integer(Class_factor)str(num_vars)## 'data.frame': 103594 obs. of 10 variables:
## $ Inflight.service : int 5 4 4 4 3 4 5 5 1 3 ...
## $ Customer.Type : int 1 2 1 1 1 1 1 1 1 2 ...
## $ Type.of.Travel : int 1 2 2 2 2 1 1 2 2 2 ...
## $ Class : int 1 2 2 2 2 3 3 2 2 3 ...
## $ Flight.Distance_log : num 6.13 5.46 7.04 6.33 5.37 ...
## $ Inflight.wifi.service : int 3 3 2 2 3 3 2 4 1 3 ...
## $ Seat.comfort : int 5 1 5 2 5 1 2 5 3 3 ...
## $ Inflight.entertainment: int 5 1 5 2 3 1 2 5 1 2 ...
## $ On.board.service : int 4 1 4 2 3 3 3 5 1 2 ...
## $ Cleanliness : int 5 1 5 2 3 1 2 4 2 2 ...
## - attr(*, "na.action")= 'omit' Named int [1:310] 214 1125 1530 2005 2109 2486 2631 3622 4042 4491 ...
## ..- attr(*, "names")= chr [1:310] "214" "1125" "1530" "2005" ...library(corrplot)
cor_matrix = cor(num_vars, use = "complete.obs")
corrplot(cor_matrix, method = "number")
Inflight.service는 On.board.service와 가장 큰 상관관계가 있다는 것을 알 수 있다. On.board.service를 Inflight.service의 예측변수로 이용해서 Polynomial model을 만들기로 한다.
ggplot(airline_train_set, aes(x=factor(On.board.service), fill=factor(Inflight.service))) +
geom_bar(position="fill") +
labs(x="On.board.service", y="Proportion", title="Proportion")
On.board.service의 만족도가 높을수록 Inflight.service역시 높은 것을
확인할 수 있다.
polynomial model : Inflight.service ~ On.board.service
#10-fold cv
ctrl = trainControl(method = "cv", number = 10)
max_degree = 10
cv_errors = rep(NA, max_degree)
model = train(Inflight.service ~ poly(On.board.service, 1, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[1] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 2, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[2] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 3, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[3] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 4, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[4] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 5, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[5] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 6, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[6] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 7, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[7] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 8, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[8] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 9, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[9] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 10, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[10] = mean(model$results$RMSE^2)# 교차 검증 오차를 그래프로 시각화
plot(1:max_degree, cv_errors, type = "b",
xlab = "Degree of Polynomial", ylab = "Cross-Validation Error",
main = "Cross-Validation Error vs. Polynomial Degree")
차수가 4부터 cv-error가 큰 폭으로 감소하는 것을 확인할 수 있다. 해당
polynomial model의 차수는 4를 선택하기로 한다.
Evaluation : polynomial model(order : 4)
poly_model = lm(Inflight.service ~ poly(On.board.service, 4) ,data = airline_train_set)
summary(poly_model)##
## Call:
## lm(formula = Inflight.service ~ poly(On.board.service, 4), data = airline_train_set)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.3230 -0.5553 0.1252 0.4899 2.4118
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.640761 0.003026 1203.24 < 2e-16 ***
## poly(On.board.service, 4)1 208.382318 0.973886 213.97 < 2e-16 ***
## poly(On.board.service, 4)2 13.213559 0.973886 13.57 < 2e-16 ***
## poly(On.board.service, 4)3 -7.927480 0.973886 -8.14 3.99e-16 ***
## poly(On.board.service, 4)4 35.495545 0.973886 36.45 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.9739 on 103589 degrees of freedom
## Multiple R-squared: 0.3138, Adjusted R-squared: 0.3137
## F-statistic: 1.184e+04 on 4 and 103589 DF, p-value: < 2.2e-16R-squared의 설명력이 낮은 이유는 예측하려는 종속변수가 카테고리 변수이기 때문임을 알 수 이다. test set으로 모델을 평가할 때, 모델의 출력을 그대로 사용하는 것이 아니라, 변환하는 과정이 필요하다.
#evaluation : test set
prediction_poly = predict(poly_model, newdata = airline_test_set)
predicted_poly_class = round(prediction_poly)actual_labels = airline_test_set$Inflight.service
predicted_labels = predicted_poly_class
predicted_labels = factor((predicted_labels), levels = c(0,1,2,3,4,5))
print(levels(predicted_labels))## [1] "0" "1" "2" "3" "4" "5"confusionMatrix(predicted_labels, factor(actual_labels))## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1 2 3 4 5
## 0 0 0 0 0 0 0
## 1 0 0 0 0 0 0
## 2 0 0 0 0 0 0
## 3 0 1197 2012 1584 1350 421
## 4 0 406 636 3227 6902 2333
## 5 0 172 190 194 1100 4167
##
## Overall Statistics
##
## Accuracy : 0.4887
## 95% CI : (0.4826, 0.4948)
## No Information Rate : 0.3612
## P-Value [Acc > NIR] : < 2.2e-16
##
## Kappa : 0.2721
##
## Mcnemar's Test P-Value : NA
##
## Statistics by Class:
##
## Class: 0 Class: 1 Class: 2 Class: 3 Class: 4 Class: 5
## Sensitivity NA 0.00000 0.0000 0.31648 0.7380 0.6021
## Specificity 1 1.00000 1.0000 0.76156 0.6008 0.9127
## Pos Pred Value NA NaN NaN 0.24132 0.5111 0.7156
## Neg Pred Value NA 0.93144 0.8904 0.82299 0.8022 0.8628
## Prevalence 0 0.06856 0.1096 0.19331 0.3612 0.2673
## Detection Rate 0 0.00000 0.0000 0.06118 0.2666 0.1609
## Detection Prevalence 0 0.00000 0.0000 0.25352 0.5216 0.2249
## Balanced Accuracy NA 0.50000 0.5000 0.53902 0.6694 0.7574Sensitivity(민감도)
- Class 0 : NAN
- Class 1, 2 : 0.0% -> 모델이 해당 클래스를 전혀 예측하지 못하고
있다.
- Class 3 : 31.65% -> 해당 클래스를 올바르게 예측한 비율이 약 32%로
낮았다.
- Class 4 : 73.80% -> 해당 클래스를 올바르게 예측한 비율이 약
74%이다.
- Class 5 : 60.21% -> 해당 클래스를 올바르게 예측한 비율이 약 60%이다.
Specificity(특이도)
- Class 0, 1, 2 : 100%
- Class 3 : 76.1%
- Class 4 : 60.0%
- Class 5 : 91.2%
result
class 4를 제외한 나머지 class에 대해서는 낮은 성능을 보였다.
Inflight.service를 예측하는 데, polynomial model은 적절않다는 결론을
내린다. 정규화 항을 추가해보기로 한다.
polynomial model : Inflight.service ~ On.board.service + regularization Term(Ridge)
airline_test_set_filtered = airline_test_set[airline_test_set$Inflight.service != 0, ]
airline_train_set_filtered = airline_train_set[airline_train_set$Inflight.service != 0, ]library(glmnet)## Loading required package: Matrix## Loaded glmnet 4.1-8x = model.matrix( ~ poly(On.board.service, 4), data = airline_train_set_filtered)[, -1]
y = as.factor(airline_train_set_filtered$Inflight.service)
ridge_model = glmnet(x, y, family = "multinomial" ,alpha = 0)# cv를 통해서 lambda 선택
cv_fit = cv.glmnet(x, y, family="multinomial", alpha=0, type.measure="class")
best_lambda = cv_fit$lambda.min print(best_lambda)## [1] 0.04341833적절한 람다값은 0.03956082 이다. 해당 값으로 모델을 fitting 하기로 한다.
y = as.integer(as.factor(airline_train_set_filtered$Inflight.service)) - 1
best_ridge_model = glmnet(x, y, alpha = 0, lambda = best_lambda)newx_test = model.matrix( ~ poly(On.board.service, 4), data = airline_test_set_filtered)[, -1]
predicted = predict(best_ridge_model, newx=newx_test, s="lambda.min", type="response")predicted_class = pmax(0, pmin(5, predicted))
final_predicted = round(predicted_class)final_predicted## [1] 4 3 3 1 1 3 4 3 1 3 2 1 4 4 3 3 3 2 4 3 4 3 1 1 2 2 4 1 4 2 2 1 3 4 4 2
## [37] 3 4 3 2 2 3 4 4 1 4 4 4 4 3 1 4 3 1 3 1 3 2 1 3 3 4 4 2 4 3 3 4 4 3 1 2
## [73] 1 3 2 4 3 4 3 1 2 1 3 2 3 1 1 4 4 3 4 4 3 1 3 1 4 3 1 2 1 4 1 4 1 2 3 2
## [109] 3 4 1 1 3 1 2 3 2 3 3 3 2 4 4 2 4 1 2 1 3 3 2 4 4 2 4 2 3 3 4 3 3 3 3 3
## [145] 1 3 3 3 4 1 1 3 2 1 4 3 1 3 3 1 4 2 4 4 1 4 1 1 4 3 4 2 3 3 4 4 3 4 3 1
## [181] 2 1 2 1 4 3 1 3 3 2 4 4 1 3 1 2 2 1 2 3 1 3 1 3 3 4 1 1 1 4 1 3 1 3 3 1
## [217] 2 3 2 4 2 3 3 3 1 1 2 3 1 3 2 4 4 3 3 2 2 3 1 3 4 2 4 4 1 3 2 2 1 4 4 3
## [253] 1 3 2 4 3 3 2 3 2 4 1 2 4 1 2 4 4 3 2 3 2 2 4 3 3 2 2 1 2 1 1 1 3 1 2 4
## [289] 4 3 1 2 2 1 2 4 2 2 2 4 2 4 4 3 2 3 4 1 4 2 1 4 4 3 3 1 3 4 4 2 2 4 2 4
## [325] 3 3 4 2 2 1 1 1 1 2 2 3 4 4 1 1 2 3 1 2 3 1 2 3 2 1 3 1 2 2 1 4 4 2 3 4
## [361] 2 2 4 2 1 3 3 1 4 2 3 1 1 1 3 4 1 1 4 3 1 1 1 3 4 3 1 3 1 4 3 4 3 3 2 3
## [397] 4 1 1 2 1 3 1 3 2 1 3 2 1 2 1 2 3 2 3 4 4 3 4 2 3 4 1 2 4 1 2 1 1 4 4 1
## [433] 4 4 3 3 4 1 4 1 2 1 2 1 3 2 3 4 2 3 3 4 1 2 1 1 1 3 3 2 1 2 2 4 1 4 4 3
## [469] 4 4 3 3 1 3 1 1 3 4 2 3 4 3 3 2 3 3 3 3 3 2 3 1 2 1 4 3 2 4 3 3 3 3 2 1
## [505] 3 4 1 4 3 3 3 1 1 1 2 1 1 3 1 1 2 2 1 3 2 3 2 3 1 3 1 4 2 2 1 3 2 3 3 1
## [541] 3 4 3 4 4 3 3 3 3 3 3 3 3 4 1 4 2 4 1 3 2 3 4 1 3 1 4 2 1 2 2 3 4 4 1 1
## [577] 2 2 1 4 3 3 2 4 3 3 2 1 3 4 2 2 1 3 1 1 1 1 4 1 2 2 3 2 2 3 3 1 3 2 2 1
## [613] 1 3 3 4 3 2 1 2 2 4 2 4 2 4 2 3 2 2 4 3 1 2 2 4 2 2 2 3 3 1 1 3 3 2 1 3
## [649] 1 2 1 1 1 1 4 3 2 3 4 3 2 2 1 1 3 2 3 4 1 1 3 3 1 4 3 2 3 3 2 3 4 2 3 4
## [685] 1 3 2 3 3 1 3 3 4 4 3 1 1 2 4 4 1 2 3 3 4 3 1 3 4 3 1 3 1 3 3 1 1 4 3 3
## [721] 3 4 4 3 2 3 3 3 4 4 2 3 4 1 1 1 1 2 3 3 3 3 4 3 3 3 1 3 3 3 4 2 4 3 1 2
## [757] 3 1 3 1 3 1 2 3 1 1 3 4 1 2 2 2 4 1 3 2 2 1 4 1 2 3 4 1 4 1 2 3 1 4 4 1
## [793] 1 1 3 2 3 3 1 2 4 1 1 4 2 1 4 3 1 3 1 1 3 1 3 1 2 2 2 2 1 2 3 3 4 1 4 4
## [829] 3 1 3 4 2 3 3 4 2 1 4 1 2 3 3 1 1 4 3 1 3 4 3 4 4 2 3 4 1 3 3 1 3 1 3 2
## [865] 3 1 3 2 2 1 2 3 4 4 4 2 3 2 4 3 2 1 3 3 2 1 4 3 1 3 1 1 1 3 3 1 4 3 4 3
## [901] 1 3 3 2 3 1 3 1 2 1 3 2 1 4 4 4 4 1 4 1 4 2 2 3 3 2 3 3 2 3 1 2 4 4 4 4
## [937] 2 2 4 3 4 3 2 4 4 2 2 3 4 1 1 4 2 2 2 3 2 3 1 4 3 2 2 2 2 1 4 1 4 3 1 1
## [973] 3 1 1 1 4 1 3 1 4 2 1 2 2 3 2 1 3 3 2 4 3 2 2 3 3 2 2 2 3 4 4 1 3 4 1 1
## [1009] 3 4 2 3 2 3 3 3 2 1 1 4 2 1 3 4 1 3 4 3 2 1 4 3 1 2 2 4 3 2 1 2 1 3 1 2
## [1045] 3 3 4 3 4 1 1 3 3 3 1 4 1 1 3 3 4 4 4 4 2 2 3 2 3 1 1 3 3 1 3 1 1 3 1 4
## [1081] 1 2 2 2 4 4 1 2 2 3 2 4 1 2 3 1 1 1 2 4 4 2 3 2 1 1 2 3 2 3 3 3 4 3 3 1
## [1117] 1 2 1 4 1 4 4 4 3 3 1 2 4 3 3 3 1 2 3 3 1 1 3 1 3 4 3 4 1 3 3 2 4 4 4 1
## [1153] 3 4 2 2 4 2 4 4 3 4 3 3 3 1 3 4 3 1 3 3 2 4 1 1 1 2 4 3 2 1 1 3 3 1 3 4
## [1189] 1 2 3 3 4 2 3 3 1 4 3 3 3 4 3 1 4 2 2 1 4 1 1 3 3 2 1 2 4 3 1 1 3 1 2 2
## [1225] 1 2 3 4 4 3 2 3 1 1 1 3 1 2 4 4 4 3 3 1 4 3 4 3 3 4 4 2 3 1 4 3 1 4 2 2
## [1261] 2 3 1 3 1 4 4 3 4 2 3 1 1 1 1 1 3 3 4 3 2 1 3 4 2 1 2 3 1 3 4 2 2 3 2 1
## [1297] 2 3 2 2 3 2 1 4 3 1 1 4 1 2 3 3 1 4 3 3 3 3 1 3 1 3 4 4 2 2 3 2 2 4 2 1
## [1333] 3 3 2 3 1 1 4 4 3 4 3 2 2 3 1 4 4 4 4 2 1 1 3 3 3 4 3 2 4 3 3 3 1 1 2 4
## [1369] 4 2 2 1 4 3 4 1 3 3 2 1 4 4 4 3 2 4 3 4 2 1 4 3 1 2 4 3 1 1 3 1 1 4 2 1
## [1405] 1 4 2 3 4 1 1 3 2 4 2 1 2 3 4 4 1 3 1 1 3 3 3 1 4 4 2 3 3 4 2 1 1 3 3 3
## [1441] 4 4 2 3 3 4 4 4 4 3 1 2 2 3 4 3 3 1 1 3 3 3 2 4 1 1 4 2 1 3 1 4 1 3 4 4
## [1477] 2 1 3 4 4 3 3 4 4 2 3 3 3 3 3 4 1 2 1 2 1 1 3 2 3 4 3 1 4 2 1 3 1 4 1 2
## [1513] 1 4 1 2 3 2 2 1 3 3 2 3 2 3 4 4 1 3 2 4 2 4 3 2 1 1 3 3 1 3 2 1 2 3 2 3
## [1549] 3 2 4 1 3 3 4 3 4 3 2 4 3 3 3 3 3 2 4 1 4 2 4 2 1 2 1 4 1 2 3 4 2 4 1 3
## [1585] 3 4 1 2 1 3 1 4 1 3 4 2 2 3 1 1 2 4 2 1 2 3 1 3 3 2 1 4 3 3 1 2 4 1 3 3
## [1621] 1 2 2 1 2 1 1 1 3 1 4 1 3 3 3 3 2 2 1 4 4 2 3 3 4 3 1 1 4 3 1 3 4 2 4 3
## [1657] 3 4 4 1 3 2 1 1 3 4 1 1 1 2 3 2 1 3 1 3 1 3 3 1 4 2 1 2 4 4 3 3 2 3 3 1
## [1693] 2 1 2 1 1 3 2 1 1 1 4 4 4 4 1 3 1 2 2 1 2 2 2 3 4 2 2 1 1 4 3 1 3 4 4 2
## [1729] 1 1 2 3 3 4 1 1 4 3 1 4 4 1 1 3 1 2 3 2 3 2 1 1 1 2 1 1 4 1 3 3 2 4 1 1
## [1765] 3 3 3 1 4 4 2 1 2 2 2 3 2 1 3 3 3 2 4 2 3 2 2 3 3 4 3 4 3 3 2 4 2 1 1 3
## [1801] 4 1 3 4 3 1 3 1 2 3 4 3 1 3 4 1 4 2 4 2 2 3 3 2 1 3 4 2 1 1 3 3 4 1 3 1
## [1837] 4 1 2 3 4 4 3 4 1 1 1 3 3 4 1 3 4 3 3 3 1 4 3 4 3 2 4 4 3 4 2 2 4 3 2 1
## [1873] 3 2 1 3 1 4 4 2 1 2 3 4 2 4 1 4 2 3 2 3 3 1 1 1 1 4 3 2 3 3 4 4 2 2 1 1
## [1909] 3 1 4 3 1 3 4 3 2 1 4 4 4 3 1 3 1 3 2 3 4 2 1 1 4 3 1 2 1 4 3 3 2 4 3 2
## [1945] 3 4 3 1 1 1 4 1 4 3 4 4 3 1 3 1 3 4 4 3 2 1 1 2 1 4 1 1 3 4 1 2 3 2 3 4
## [1981] 1 3 3 3 4 1 2 1 4 2 2 2 1 4 3 2 1 3 1 1 2 3 1 1 4 3 4 2 4 4 3 3 4 3 3 3
## [2017] 2 2 3 3 1 1 1 3 4 1 4 4 4 4 1 2 3 3 3 4 1 1 2 3 2 4 4 3 3 3 3 4 3 3 1 3
## [2053] 4 4 4 3 2 2 3 2 4 1 3 4 3 2 4 1 2 3 1 1 3 1 1 4 2 3 2 1 4 2 3 2 1 1 2 3
## [2089] 2 1 3 4 3 4 3 4 1 2 3 1 2 4 4 2 1 4 2 3 2 4 2 2 3 1 3 3 2 3 4 3 3 1 2 1
## [2125] 3 1 2 1 4 2 4 3 4 4 1 4 3 3 2 1 3 2 1 2 4 4 1 1 2 4 2 1 3 2 3 2 4 3 4 4
## [2161] 3 1 4 4 2 3 2 1 2 4 2 1 1 4 1 3 1 2 2 3 3 2 1 4 2 1 2 4 2 1 2 2 4 3 2 3
## [2197] 1 3 2 1 2 1 1 4 3 2 1 3 1 2 2 4 4 4 3 2 1 4 3 2 3 2 3 1 2 3 3 3 4 2 4 4
## [2233] 4 2 4 1 3 2 4 3 2 1 1 1 2 2 1 2 2 2 3 3 1 3 2 4 4 2 2 3 4 4 4 4 3 3 3 2
## [2269] 2 3 2 2 1 1 3 4 4 3 3 1 1 2 4 4 1 1 3 3 2 3 1 3 4 2 3 1 2 1 1 2 2 1 2 4
## [2305] 2 4 3 1 2 4 4 2 4 3 3 1 4 4 2 3 3 2 4 3 4 1 3 1 2 2 2 2 2 3 1 3 4 3 3 3
## [2341] 4 2 2 3 3 2 3 1 1 1 3 1 4 4 1 3 4 2 2 4 2 3 1 2 1 1 3 1 1 3 3 4 3 3 1 3
## [2377] 3 3 4 4 1 3 3 1 1 1 4 3 3 3 1 1 3 2 1 4 4 4 4 4 4 4 3 1 4 4 2 2 1 2 4 3
## [2413] 1 2 1 2 4 1 3 4 3 4 1 3 1 4 4 2 3 2 3 4 3 1 4 4 1 2 4 3 4 2 4 1 4 2 3 4
## [2449] 1 3 2 3 4 2 1 2 2 4 4 3 2 2 1 2 1 3 3 3 1 4 1 4 3 2 3 3 1 4 3 3 1 2 2 4
## [2485] 1 2 1 1 2 4 1 4 3 1 3 1 1 3 1 4 4 4 3 2 1 4 3 4 3 4 3 4 1 3 3 4 4 3 1 1
## [2521] 2 3 2 1 3 1 3 3 3 3 1 4 4 3 3 4 3 1 1 2 3 4 3 1 4 2 3 3 3 3 4 3 2 4 1 3
## [2557] 4 1 3 1 3 2 2 1 4 4 3 1 3 2 2 1 3 2 1 3 1 2 3 3 3 1 3 1 1 1 2 3 3 4 2 2
## [2593] 1 1 3 1 4 3 2 3 1 2 4 1 2 3 1 3 1 2 2 2 1 2 2 3 3 3 3 3 2 2 1 1 1 2 2 3
## [2629] 3 1 4 3 4 3 3 1 1 4 4 1 2 4 2 3 1 3 1 3 1 4 3 1 1 1 4 1 1 1 1 2 3 1 4 4
## [2665] 3 3 2 1 1 3 4 2 1 3 2 4 1 2 4 2 1 2 2 4 2 3 2 2 4 1 1 4 3 1 3 2 1 4 3 3
## [2701] 1 3 3 2 2 3 4 3 1 1 2 2 3 3 3 4 2 3 1 3 4 1 3 2 4 1 1 3 1 2 2 4 4 1 4 2
## [2737] 1 3 3 4 3 1 3 2 4 1 3 3 3 3 3 1 2 2 2 1 3 3 1 1 3 1 2 4 3 2 2 3 3 4 2 4
## [2773] 3 2 3 2 2 1 2 3 2 1 3 4 2 2 2 2 2 2 1 3 1 3 4 2 2 2 4 1 2 4 3 1 2 1 3 3
## [2809] 1 2 4 1 2 3 3 1 3 3 1 2 1 1 3 4 4 2 3 4 2 2 4 2 2 4 3 3 1 4 4 4 3 1 4 2
## [2845] 1 1 2 3 3 2 4 1 3 1 2 2 3 3 4 3 3 2 1 4 4 4 2 3 3 1 4 3 1 4 1 2 2 1 3 4
## [2881] 2 4 3 4 1 4 2 2 3 1 1 3 3 4 3 2 1 4 3 1 4 1 1 3 4 3 2 3 3 4 3 1 3 1 3 3
## [2917] 4 4 1 3 1 3 4 4 4 1 2 4 2 4 2 3 1 2 1 1 2 1 1 1 2 1 2 4 1 3 3 3 1 4 1 2
## [2953] 1 2 2 2 4 4 4 4 2 4 2 2 3 3 3 2 2 2 4 3 4 3 3 1 1 4 2 1 3 4 4 3 2 4 3 2
## [2989] 1 3 1 3 3 3 4 4 4 2 2 2 1 4 2 3 4 3 1 1 1 4 1 2 1 1 1 3 1 3 1 1 1 3 2 2
## [3025] 4 3 3 3 1 3 4 4 4 1 3 2 3 3 4 2 1 2 3 3 4 1 1 3 3 3 4 2 1 4 2 3 3 1 2 2
## [3061] 1 3 2 2 3 2 2 3 2 2 3 3 3 3 3 3 3 3 4 1 4 4 3 1 3 3 3 3 2 3 4 4 1 3 4 1
## [3097] 2 3 1 2 4 3 3 1 3 3 3 1 3 3 1 1 4 2 3 1 4 2 2 2 3 2 2 1 2 3 4 4 4 1 2 3
## [3133] 1 3 2 1 1 2 3 1 3 4 1 2 1 1 1 2 3 4 3 1 1 4 2 4 3 3 3 3 3 4 3 4 1 1 3 2
## [3169] 2 1 1 3 4 2 1 1 3 3 3 4 3 3 3 2 1 1 2 2 1 2 3 2 3 4 1 3 1 2 1 3 1 3 3 2
## [3205] 3 3 1 1 1 3 3 2 1 3 4 2 1 3 2 4 4 4 4 1 3 3 4 3 4 1 1 1 1 3 3 2 2 4 1 4
## [3241] 3 3 3 2 4 4 3 1 2 3 2 2 2 1 4 3 2 2 4 1 1 1 2 3 3 2 3 2 4 3 1 2 1 1 1 2
## [3277] 2 1 1 1 2 1 3 3 1 4 4 3 3 1 3 1 1 1 3 4 3 3 1 3 1 1 4 2 1 2 3 4 4 1 2 1
## [3313] 1 2 1 2 3 1 3 2 4 2 3 4 3 3 4 2 4 3 4 3 2 3 1 2 4 1 1 1 2 1 2 4 3 1 4 3
## [3349] 3 4 1 2 2 2 2 3 3 2 3 2 4 3 1 2 4 1 4 1 4 1 1 3 3 4 3 3 1 3 1 4 1 1 1 3
## [3385] 1 1 4 1 3 1 3 4 1 4 1 2 4 4 3 2 2 3 3 1 2 3 4 3 4 1 2 3 4 3 1 1 4 1 1 3
## [3421] 4 3 1 1 1 1 4 1 3 3 3 3 1 2 4 1 1 3 3 3 3 2 4 3 3 3 2 3 3 2 3 4 1 1 1 4
## [3457] 1 2 4 2 3 3 3 4 1 1 1 3 4 3 4 2 3 4 3 2 3 3 1 1 2 2 1 3 1 2 3 3 4 1 3 3
## [3493] 3 4 2 1 4 1 3 4 1 4 1 3 1 4 3 2 2 3 1 3 1 3 3 1 3 4 4 3 1 3 3 3 1 4 3 3
## [3529] 1 3 1 3 3 4 1 4 3 2 4 1 2 3 1 2 3 2 2 4 3 2 1 4 2 4 1 4 2 3 1 1 3 1 2 2
## [3565] 3 4 4 2 4 4 3 2 4 4 1 2 3 3 4 4 4 3 1 4 3 4 2 1 2 3 1 3 3 1 1 3 2 1 3 3
## [3601] 3 3 2 4 1 3 2 4 3 2 1 4 2 2 2 1 1 4 1 2 2 3 4 4 4 2 4 2 1 4 2 3 2 3 4 1
## [3637] 1 1 2 3 1 1 4 1 3 1 1 1 3 1 2 2 2 3 3 1 1 3 3 4 2 1 1 3 3 3 4 2 1 3 1 4
## [3673] 3 2 1 1 3 1 4 2 3 3 1 3 3 3 4 3 2 3 4 2 4 1 1 3 3 3 1 2 1 3 1 1 3 2 3 1
## [3709] 1 3 4 1 3 2 3 2 1 4 1 2 2 2 3 4 1 3 3 4 4 1 2 3 1 2 1 4 3 1 1 3 3 1 1 2
## [3745] 2 3 3 3 1 3 4 1 4 2 3 1 2 3 2 3 3 1 3 4 1 2 2 3 1 2 4 1 4 2 3 2 3 1 1 1
## [3781] 4 2 2 1 1 4 2 1 2 2 2 4 1 2 2 2 1 3 3 2 1 2 4 3 1 3 3 3 2 3 4 3 3 1 2 4
## [3817] 2 3 2 1 2 1 4 3 4 2 2 3 4 1 4 3 4 1 3 1 1 1 3 4 1 2 4 3 2 3 4 2 3 1 2 2
## [3853] 1 3 1 4 4 3 1 4 1 4 1 3 2 3 4 1 4 1 4 1 1 1 3 4 3 1 3 1 3 1 1 3 3 3 3 2
## [3889] 2 4 4 3 4 3 3 2 2 3 2 3 4 2 2 3 4 2 3 3 3 1 4 4 4 2 4 2 2 4 4 4 4 4 3 3
## [3925] 3 3 4 1 1 1 3 3 4 2 4 2 1 1 2 2 3 3 3 1 4 3 3 1 3 4 3 1 1 1 1 1 3 2 3 3
## [3961] 2 3 3 4 4 3 2 3 1 1 3 4 3 2 2 3 2 2 3 4 4 2 4 3 1 4 1 1 3 3 1 4 3 4 3 3
## [3997] 3 4 3 3 4 2 3 1 1 4 2 3 1 3 2 3 4 1 3 1 1 1 1 3 3 3 4 2 1 1 1 3 2 2 2 2
## [4033] 1 3 4 1 3 3 3 4 4 3 3 2 3 3 1 2 1 3 4 2 4 4 1 1 3 4 1 3 3 1 4 1 3 2 3 3
## [4069] 2 1 1 4 1 1 3 3 1 2 3 2 4 3 3 3 1 2 2 1 1 4 1 4 1 1 4 4 4 4 2 4 3 1 3 3
## [4105] 3 1 2 3 4 4 4 1 3 4 2 3 3 1 1 2 3 3 3 3 1 1 3 1 3 1 3 2 4 2 4 4 3 2 4 3
## [4141] 1 2 4 2 4 3 3 1 1 3 1 2 1 3 1 1 4 3 1 4 1 3 1 1 2 1 2 4 4 3 1 4 3 2 2 3
## [4177] 1 4 3 2 1 4 2 2 1 4 4 4 1 4 3 4 1 2 3 3 2 3 4 4 3 2 4 2 4 1 3 1 3 3 4 4
## [4213] 2 1 2 3 1 3 3 2 1 2 1 4 4 1 2 2 4 3 3 2 2 4 3 3 2 2 1 1 2 4 1 3 4 4 2 2
## [4249] 2 1 2 2 2 1 2 1 3 4 4 2 4 3 3 4 1 2 4 1 4 2 2 2 1 1 3 2 4 4 4 3 1 2 1 1
## [4285] 2 3 3 2 2 1 1 4 3 2 4 4 3 2 3 1 2 2 1 3 2 4 4 1 4 4 3 2 2 4 1 3 2 3 4 1
## [4321] 4 3 4 2 4 1 3 3 4 3 3 1 4 3 3 3 3 3 1 4 3 2 2 4 3 3 4 1 2 4 3 4 4 3 1 3
## [4357] 2 2 3 1 3 4 4 1 2 4 2 4 3 1 4 3 3 3 2 1 3 3 1 3 1 4 2 2 4 2 1 4 3 3 1 1
## [4393] 2 3 3 2 1 1 4 4 1 4 2 1 1 4 3 3 4 4 1 4 2 1 1 4 2 2 3 1 4 4 4 2 4 3 4 4
## [4429] 1 1 1 2 3 3 2 1 4 4 3 4 1 1 1 1 1 1 3 3 3 4 4 3 4 4 4 3 3 2 3 1 1 3 3 3
## [4465] 1 3 3 3 3 3 4 1 3 2 4 3 3 3 2 2 2 2 2 2 3 3 2 1 1 1 4 4 3 3 3 4 1 3 3 4
## [4501] 1 1 3 3 2 1 1 2 3 3 1 4 4 1 3 1 2 4 1 3 3 4 1 2 1 1 2 1 4 3 2 2 3 2 2 4
## [4537] 2 3 2 4 3 2 1 2 1 1 2 1 3 3 3 4 4 4 4 3 4 2 3 4 2 3 1 4 3 4 2 3 4 3 4 4
## [4573] 3 3 4 3 2 1 2 3 1 1 1 4 1 4 3 1 4 3 3 4 3 3 1 2 3 1 4 2 3 1 4 1 2 3 2 1
## [4609] 3 3 2 3 1 3 1 3 2 3 1 3 2 1 2 3 2 2 1 2 1 2 3 2 4 2 4 4 4 2 4 4 3 3 3 2
## [4645] 2 1 2 1 1 2 4 3 2 3 1 2 4 3 3 3 1 4 3 2 2 3 3 1 1 3 3 1 1 1 4 3 4 1 4 4
## [4681] 1 4 1 4 4 3 3 4 2 2 4 3 1 3 1 3 3 4 4 1 2 2 2 4 1 1 2 3 1 1 3 1 1 1 4 4
## [4717] 2 4 2 2 4 2 1 4 3 3 1 1 2 1 1 4 3 3 2 1 3 3 3 2 3 3 3 4 4 3 2 3 4 1 3 2
## [4753] 2 3 1 3 1 2 2 2 3 2 2 2 3 4 2 2 3 3 4 3 2 3 2 1 1 1 3 1 1 4 2 4 1 4 4 4
## [4789] 3 2 4 1 2 2 2 2 2 2 1 3 2 4 1 2 2 3 3 4 2 3 1 2 4 4 3 4 4 1 3 1 1 4 2 1
## [4825] 1 3 2 3 4 4 2 1 3 1 4 1 2 4 2 2 4 1 1 4 1 3 4 4 3 1 4 3 3 3 1 2 1 3 1 2
## [4861] 1 3 4 4 1 2 4 3 1 3 4 2 3 3 3 4 2 4 2 1 3 4 1 2 4 4 3 3 4 3 2 3 4 2 2 4
## [4897] 4 3 4 2 4 1 1 3 4 1 3 3 3 3 3 1 1 2 1 3 1 2 2 2 1 4 3 4 3 3 4 3 1 2 1 4
## [4933] 2 2 2 4 2 2 4 2 2 3 3 3 3 3 4 3 2 2 1 3 3 4 3 2 1 4 3 1 1 3 1 1 1 3 1 2
## [4969] 3 1 2 3 1 2 2 2 2 4 4 4 4 1 4 4 4 3 4 4 2 3 3 4 3 1 2 2 3 3 4 2 3 3 4 4
## [5005] 1 4 4 2 2 1 2 2 4 3 1 3 3 4 4 4 2 1 2 1 3 4 3 4 3 2 4 4 2 4 4 3 1 1 3 1
## [5041] 3 4 1 2 3 3 3 3 3 1 1 2 1 1 1 4 3 1 3 2 3 3 2 3 2 1 3 3 4 3 2 2 3 1 3 3
## [5077] 3 4 3 3 1 3 1 2 3 2 2 3 3 1 3 4 2 2 3 2 1 2 4 2 2 4 3 1 2 4 4 2 1 2 3 2
## [5113] 3 1 2 1 1 3 4 3 4 2 3 3 3 4 1 4 1 3 3 4 3 1 4 3 4 4 4 2 1 2 2 2 3 3 3 2
## [5149] 3 4 3 3 3 3 1 1 2 1 4 2 3 1 2 4 1 1 2 2 1 3 2 3 1 4 3 4 3 3 3 4 4 4 3 3
## [5185] 3 1 3 4 1 3 2 3 1 4 1 2 2 1 3 2 3 4 2 1 2 4 4 1 1 4 2 4 1 4 3 2 3 3 4 1
## [5221] 1 4 2 3 3 3 2 2 3 3 3 1 1 3 4 3 1 2 2 2 2 2 1 3 4 4 2 3 3 3 1 4 3 2 4 2
## [5257] 3 1 4 2 3 4 2 1 1 3 1 1 2 2 1 2 1 2 3 2 2 3 4 2 2 4 3 1 1 3 3 3 2 1 3 4
## [5293] 3 2 3 3 4 1 3 4 4 3 1 4 4 1 2 2 3 3 4 4 2 1 3 2 3 4 4 2 2 2 3 3 1 2 4 2
## [5329] 4 1 4 1 2 2 3 3 3 1 3 4 1 1 4 2 2 1 3 3 1 4 3 2 3 1 1 3 4 3 3 3 4 1 3 4
## [5365] 3 3 1 1 1 4 4 4 1 4 4 2 4 1 2 3 4 1 3 3 2 1 1 4 2 3 1 1 4 3 3 3 2 2 4 2
## [5401] 2 2 2 3 4 3 3 1 3 1 4 1 3 3 3 4 2 2 1 3 1 1 2 4 4 4 1 2 1 1 2 2 1 4 2 2
## [5437] 4 1 3 4 3 4 1 4 4 2 4 4 2 1 4 4 1 2 4 2 2 2 4 3 4 4 1 4 4 3 4 1 2 3 2 4
## [5473] 2 1 3 4 2 1 2 4 1 1 2 4 2 3 1 1 1 3 1 1 4 4 2 4 1 1 4 2 1 4 4 4 4 3 3 4
## [5509] 4 2 1 1 3 3 4 1 3 1 4 4 1 3 3 2 1 3 3 1 3 4 4 2 2 3 2 1 3 2 3 2 3 4 3 2
## [5545] 1 4 3 1 2 4 3 1 3 2 1 3 4 3 3 3 3 1 4 1 3 4 2 1 1 4 2 4 2 3 3 3 3 2 2 3
## [5581] 4 4 4 2 3 4 2 1 3 2 2 2 1 3 3 2 2 1 2 3 2 3 4 4 1 3 3 1 1 4 1 2 4 4 1 2
## [5617] 1 3 4 2 3 3 2 1 2 2 2 1 3 3 3 4 4 2 3 2 4 4 4 4 4 1 2 3 3 1 4 4 4 3 1 3
## [5653] 4 4 2 3 4 4 4 1 1 2 1 4 4 3 1 1 2 2 1 1 1 4 3 1 4 4 3 2 2 3 4 1 3 3 1 3
## [5689] 3 1 3 1 2 1 1 3 4 3 4 2 3 3 4 4 3 4 3 2 4 4 1 4 3 1 4 1 1 4 1 2 1 1 4 4
## [5725] 3 1 4 1 3 2 3 1 4 3 3 4 2 1 2 3 2 2 2 2 1 2 4 1 1 3 4 3 3 2 3 3 4 3 3 3
## [5761] 3 1 1 2 3 3 2 3 3 3 3 2 4 2 4 4 2 3 1 3 4 3 1 2 4 1 4 3 1 1 2 3 2 2 3 4
## [5797] 1 1 1 3 3 1 3 4 4 4 4 3 2 4 1 1 4 4 4 3 4 3 3 1 3 1 4 4 3 3 3 1 1 3 1 2
## [5833] 3 3 1 2 3 3 2 4 3 2 4 2 4 2 4 3 3 2 3 1 3 3 3 2 2 4 2 3 4 3 3 2 1 1 4 3
## [5869] 4 2 4 4 1 3 2 3 1 3 3 1 4 3 1 3 2 2 1 2 4 4 3 4 4 3 1 1 3 3 4 3 1 3 1 3
## [5905] 4 3 4 3 4 1 1 3 1 3 2 4 3 1 3 2 1 4 4 2 4 3 3 2 4 4 3 1 1 3 2 3 2 3 2 1
## [5941] 4 1 1 3 3 2 2 3 3 1 1 1 2 3 4 1 4 3 3 2 2 1 1 4 4 4 4 2 1 1 2 3 2 4 1 1
## [5977] 3 3 3 1 4 2 2 2 2 3 1 2 3 1 2 4 4 3 3 3 2 1 2 3 1 1 2 2 2 4 4 2 3 2 2 4
## [6013] 1 3 4 2 3 1 4 3 4 4 3 2 1 1 2 3 4 1 1 3 2 4 2 1 1 1 3 2 1 4 1 3 4 4 3 4
## [6049] 1 1 4 1 1 4 1 1 3 4 3 1 3 2 2 1 3 2 3 2 1 3 2 1 4 1 3 3 1 1 4 3 4 4 2 3
## [6085] 3 3 1 1 1 1 4 4 4 3 1 3 2 4 3 1 3 4 1 3 2 3 1 3 4 3 1 3 2 4 2 1 3 1 3 2
## [6121] 3 1 1 2 4 3 1 3 1 4 1 3 3 3 3 4 2 3 3 1 1 4 4 4 2 3 2 2 1 1 3 2 3 1 1 1
## [6157] 1 4 1 2 1 3 2 1 2 4 4 3 2 3 1 4 4 2 4 4 3 3 1 1 2 4 4 3 2 4 4 3 1 2 2 4
## [6193] 3 3 3 2 2 3 4 4 3 1 1 2 4 3 1 1 3 3 3 4 3 1 3 4 4 3 2 2 2 4 2 4 1 2 3 3
## [6229] 1 3 1 2 4 2 2 1 1 3 2 3 1 1 4 3 2 4 4 2 1 2 2 3 4 2 3 2 2 3 2 1 4 1 3 2
## [6265] 3 1 1 3 2 4 2 2 3 3 1 3 3 2 4 4 1 2 1 1 4 2 1 2 4 1 1 4 4 1 1 1 4 3 2 1
## [6301] 3 3 2 3 3 1 2 3 2 3 3 3 3 1 2 2 1 2 1 1 3 1 3 4 2 1 4 4 1 4 2 4 4 2 4 1
## [6337] 3 4 1 4 1 2 4 3 1 3 3 3 1 3 4 3 2 3 2 2 3 1 2 3 1 4 1 3 4 2 3 1 3 4 1 4
## [6373] 2 4 4 3 3 4 1 3 2 2 2 4 3 4 1 3 3 1 1 1 2 3 4 3 2 4 2 1 1 4 3 2 1 4 1 3
## [6409] 3 4 1 1 2 4 3 2 2 3 3 1 2 2 1 2 3 4 4 3 4 2 1 3 1 4 4 1 3 2 3 1 3 3 2 3
## [6445] 1 1 1 4 3 3 1 3 4 3 3 1 1 4 4 3 1 4 2 3 2 1 3 1 3 3 4 3 3 3 3 3 4 2 4 3
## [6481] 2 4 1 3 1 2 1 4 2 3 4 4 3 2 2 1 3 3 4 1 3 1 4 3 2 4 4 3 4 4 3 3 2 1 3 2
## [6517] 3 3 1 2 3 3 4 3 1 4 4 4 2 4 3 4 1 2 2 1 3 2 2 2 3 3 4 3 4 1 2 2 3 4 4 4
## [6553] 3 3 2 3 1 2 1 3 4 2 3 2 4 2 3 1 3 3 1 4 4 1 1 3 2 4 1 3 1 1 4 3 3 4 2 4
## [6589] 4 2 2 4 4 4 4 3 2 1 3 1 1 2 3 3 1 2 2 4 2 1 1 1 4 3 1 3 1 2 2 2 3 4 2 1
## [6625] 4 3 3 1 3 1 3 1 1 4 2 4 4 2 4 3 4 2 2 4 4 3 1 2 4 2 2 3 3 3 2 3 2 3 1 3
## [6661] 1 1 4 2 1 3 3 1 1 4 1 4 3 2 1 2 1 2 1 2 2 4 1 4 4 3 2 1 4 2 4 1 1 3 1 2
## [6697] 3 3 1 4 4 1 4 1 4 3 3 1 3 4 3 3 1 4 3 4 3 4 4 4 1 1 1 3 3 4 1 3 4 2 1 3
## [6733] 4 2 1 4 1 3 1 3 4 4 4 1 2 1 1 2 4 4 3 4 1 4 3 1 2 4 3 2 2 4 3 3 1 4 2 4
## [6769] 3 4 4 2 4 2 2 2 4 2 4 2 3 4 3 1 4 2 2 3 1 3 4 3 1 3 1 1 2 1 2 4 3 1 1 4
## [6805] 2 2 2 4 3 3 3 3 3 2 3 2 3 4 3 4 2 1 1 4 2 3 1 2 1 1 2 1 1 3 4 4 3 2 4 3
## [6841] 3 2 4 4 2 3 3 3 2 2 1 3 1 2 4 3 2 3 4 4 2 2 3 3 2 1 3 3 2 3 3 2 1 1 3 1
## [6877] 1 3 4 4 3 1 4 2 3 3 2 2 1 1 1 3 1 2 4 2 3 2 4 2 1 1 2 4 3 1 3 2 3 2 4 1
## [6913] 2 1 2 4 4 3 1 3 2 3 3 3 4 1 2 3 1 4 4 2 4 1 2 1 3 3 3 3 2 2 2 3 4 4 3 3
## [6949] 2 1 1 4 2 2 2 2 1 4 2 2 4 1 3 4 4 2 3 3 3 2 2 4 4 2 2 2 1 1 3 2 2 1 1 2
## [6985] 2 1 2 2 4 4 4 1 3 1 1 4 4 3 3 4 1 3 4 3 4 4 1 3 4 3 2 3 1 1 3 2 3 3 2 2
## [7021] 1 3 3 1 3 1 2 3 1 1 3 2 2 1 1 4 1 4 3 3 3 2 3 3 3 3 1 2 4 3 4 1 4 3 1 2
## [7057] 3 2 2 3 3 3 3 2 2 3 4 2 2 1 1 2 1 4 2 1 1 3 3 3 3 4 3 3 3 3 1 1 1 1 1 1
## [7093] 4 4 2 3 3 2 1 4 1 4 3 4 1 3 3 3 3 1 3 3 3 3 2 1 1 3 3 2 3 1 3 1 4 3 2 1
## [7129] 1 1 4 4 3 3 1 4 2 4 3 2 3 4 4 1 4 3 4 2 4 3 2 4 3 1 3 3 3 1 2 4 3 2 1 1
## [7165] 4 3 2 3 1 1 2 1 2 1 2 1 1 2 4 2 1 4 1 2 2 1 1 4 2 1 4 4 1 2 3 3 4 1 4 3
## [7201] 3 1 4 3 1 4 2 4 2 4 1 4 2 3 3 4 1 1 4 2 4 3 2 1 3 1 2 1 4 2 3 4 2 3 1 3
## [7237] 1 1 4 4 1 1 3 2 3 1 2 4 4 4 2 3 1 1 2 4 3 3 3 2 3 4 3 1 3 1 2 3 1 2 3 1
## [7273] 3 4 4 2 2 1 3 2 1 4 2 3 1 3 4 3 1 4 1 2 3 3 3 3 4 1 2 1 3 3 2 4 3 1 3 3
## [7309] 4 4 4 4 1 3 4 1 1 2 3 2 3 2 4 3 2 1 1 3 3 1 3 3 3 3 1 2 2 2 3 2 2 1 2 2
## [7345] 2 1 4 1 4 2 3 3 4 2 1 1 4 4 4 1 1 3 3 2 3 2 3 3 3 3 3 2 3 3 4 4 4 4 3 2
## [7381] 1 3 3 1 2 1 1 3 4 1 2 3 2 1 2 3 3 4 4 4 4 1 1 1 3 2 3 3 1 4 2 3 1 4 2 2
## [7417] 2 1 4 1 3 1 3 1 1 2 2 2 1 2 3 2 1 4 4 3 2 3 1 3 3 2 2 1 3 2 3 3 1 2 3 3
## [7453] 1 3 1 2 3 3 3 3 4 4 3 4 2 1 4 3 4 2 2 1 3 3 1 4 2 1 4 1 1 1 3 1 1 4 3 4
## [7489] 3 3 2 2 3 3 3 4 4 4 2 4 3 1 1 3 1 2 2 1 4 1 4 2 3 3 3 2 4 4 2 3 2 2 4 1
## [7525] 1 1 1 4 3 4 1 3 3 4 2 4 2 1 2 1 3 3 1 4 1 3 1 4 1 4 3 1 2 3 3 3 4 3 3 3
## [7561] 2 2 4 4 1 1 1 1 4 3 2 1 3 1 2 3 1 3 1 3 1 1 3 1 2 4 3 4 2 4 4 3 2 4 3 2
## [7597] 4 2 1 1 4 4 1 1 1 3 2 4 4 1 1 4 1 1 2 4 4 4 2 3 3 4 3 1 2 3 4 1 1 2 1 3
## [7633] 1 1 4 3 3 3 3 4 3 4 1 2 4 1 4 1 1 2 3 2 1 2 4 3 4 3 1 1 1 4 4 4 3 2 2 3
## [7669] 3 1 1 1 3 1 3 4 2 1 2 3 3 1 2 1 3 3 4 3 2 4 3 1 1 4 4 3 4 1 1 2 4 3 4 4
## [7705] 2 2 4 4 3 4 2 4 2 3 2 1 1 3 3 2 3 2 3 3 3 3 1 3 3 1 1 1 3 4 1 1 1 4 3 1
## [7741] 3 3 4 4 4 3 1 4 2 3 3 4 4 4 2 4 4 4 4 2 4 2 1 1 3 2 3 2 4 3 3 2 2 2 2 1
## [7777] 3 4 4 2 4 1 4 3 3 1 2 1 4 1 2 3 3 2 4 4 3 4 2 3 4 3 1 4 2 3 1 4 2 4 1 4
## [7813] 2 1 2 2 3 1 3 1 1 4 3 4 1 4 3 1 1 3 2 2 1 3 2 4 3 2 2 4 1 1 2 4 3 3 3 4
## [7849] 3 1 1 4 4 1 3 2 1 4 2 3 4 1 1 2 3 1 4 3 2 4 2 1 3 3 3 2 2 2 2 3 3 2 3 2
## [7885] 2 3 3 2 3 2 1 2 1 3 1 2 1 4 3 1 4 3 1 3 2 2 1 4 3 4 2 3 3 2 3 4 3 4 2 4
## [7921] 3 4 3 1 2 1 1 1 2 1 3 4 3 2 4 1 2 1 4 4 1 2 4 2 3 3 2 3 1 3 3 1 2 1 1 1
## [7957] 4 4 2 2 1 3 4 1 3 1 2 4 1 1 3 2 1 3 1 3 3 2 3 1 1 4 3 4 2 4 1 4 2 2 1 4
## [7993] 4 4 3 1 4 2 3 1 2 2 4 2 4 2 1 4 4 3 3 3 4 3 1 1 3 2 3 3 3 1 3 3 1 2 2 3
## [8029] 1 1 2 1 4 1 3 3 2 1 3 4 2 4 3 3 1 3 2 1 1 3 3 3 4 1 3 4 4 3 4 2 1 1 1 1
## [8065] 2 4 1 3 2 1 1 3 4 1 3 1 3 4 3 1 2 3 1 3 1 3 4 3 4 3 4 3 1 2 1 3 4 3 3 2
## [8101] 2 1 2 2 2 4 4 1 3 3 2 3 3 1 4 1 4 1 1 2 2 3 3 1 1 4 3 4 2 3 3 1 3 3 3 2
## [8137] 3 3 4 4 3 3 4 3 1 4 1 2 1 2 1 1 3 3 2 4 1 2 2 2 2 2 4 4 2 4 4 1 1 2 1 3
## [8173] 3 1 2 4 3 3 4 4 2 4 3 2 1 3 2 3 3 3 2 1 3 1 4 3 1 1 3 1 1 3 1 3 3 3 3 2
## [8209] 4 4 4 4 1 4 2 1 4 2 3 4 3 3 2 3 1 1 1 1 4 3 4 3 1 1 4 2 4 2 1 4 1 1 2 3
## [8245] 2 3 4 3 2 3 4 2 1 4 2 1 2 3 4 4 1 1 4 2 2 4 4 2 2 3 4 3 1 2 3 1 4 2 3 3
## [8281] 2 4 4 3 1 4 1 1 3 2 2 4 4 2 2 1 3 2 3 3 3 3 3 4 2 4 1 4 2 3 1 3 1 4 2 1
## [8317] 4 4 3 1 3 2 1 3 3 4 1 2 1 3 4 2 2 3 3 4 3 3 2 4 1 2 1 2 3 2 1 4 4 2 3 3
## [8353] 3 3 1 1 3 1 2 3 4 4 2 4 3 3 1 4 2 2 1 3 3 1 3 2 1 3 2 4 2 1 1 1 3 1 4 1
## [8389] 2 2 3 1 3 4 3 1 2 3 3 3 3 3 3 3 2 2 3 1 3 2 1 1 2 3 2 2 2 1 2 4 2 4 4 3
## [8425] 2 2 4 3 4 3 1 4 4 4 1 1 4 1 4 4 1 4 4 2 2 2 1 3 4 4 3 3 1 1 3 3 4 1 1 1
## [8461] 2 2 3 2 2 1 2 3 1 1 2 2 3 3 4 4 1 3 3 1 4 1 3 2 3 2 2 4 4 3 3 1 1 4 2 2
## [8497] 2 2 4 1 4 2 4 4 1 3 1 2 1 1 3 3 2 3 2 4 4 3 1 3 4 3 3 2 3 4 3 3 4 3 4 4
## [8533] 1 4 4 2 2 4 2 3 1 3 3 3 4 2 4 3 1 2 1 4 4 4 3 4 2 1 1 4 4 3 3 2 4 1 1 1
## [8569] 1 4 2 1 2 3 4 1 2 1 2 4 2 1 1 4 3 1 2 4 1 2 4 1 1 1 2 3 2 1 3 1 1 3 2 2
## [8605] 1 2 1 3 1 3 3 1 2 3 2 3 4 4 3 3 4 4 4 2 2 3 1 3 3 4 4 4 4 3 2 1 4 1 4 1
## [8641] 1 3 2 4 1 2 2 3 4 2 1 1 4 1 2 1 2 1 1 3 3 3 4 3 3 4 3 4 4 4 2 2 1 1 2 3
## [8677] 4 4 3 3 1 3 2 3 1 4 3 2 2 1 2 1 2 4 2 1 4 1 4 2 3 3 1 1 3 2 3 3 2 3 2 2
## [8713] 4 3 2 1 3 1 1 2 4 3 4 3 3 1 3 3 1 4 3 2 4 2 4 2 1 3 4 2 3 2 1 1 4 2 3 4
## [8749] 1 2 4 4 3 4 4 1 1 1 2 2 3 1 2 1 4 2 1 3 3 3 2 1 2 4 4 3 2 2 1 1 4 1 3 3
## [8785] 2 4 3 3 3 1 2 3 3 3 2 3 3 1 2 3 2 4 3 3 3 4 4 1 3 1 1 1 1 1 3 1 3 1 3 2
## [8821] 1 2 2 2 3 4 1 3 1 1 4 1 3 3 4 3 4 4 4 2 3 1 1 2 1 2 1 1 1 4 1 3 1 1 2 1
## [8857] 2 1 2 4 1 2 1 1 1 1 3 2 4 3 3 1 1 2 3 3 1 4 2 1 4 4 1 2 2 4 3 4 3 3 2 1
## [8893] 3 1 3 2 2 3 1 2 4 1 2 1 3 1 4 4 3 3 3 3 1 1 1 4 2 3 3 2 4 2 3 3 2 4 1 3
## [8929] 2 2 1 2 1 4 1 2 3 4 1 4 2 1 4 1 3 1 3 1 2 1 3 4 3 1 4 3 4 1 4 1 4 3 4 4
## [8965] 3 4 2 4 1 2 3 1 3 4 3 2 3 1 4 1 2 2 3 3 1 3 2 3 1 1 2 3 1 1 3 3 2 2 4 4
## [9001] 2 4 3 2 1 2 4 3 1 3 2 3 4 3 1 1 3 1 3 4 3 3 1 1 3 3 3 1 1 3 3 1 4 2 3 3
## [9037] 2 3 3 1 4 3 1 2 3 2 1 4 3 4 4 4 3 1 2 3 4 3 3 4 3 2 4 3 1 3 3 2 1 2 3 1
## [9073] 1 3 2 1 1 4 2 3 3 3 3 2 1 3 4 3 2 1 2 1 2 3 2 1 4 2 1 3 2 1 4 3 4 1 2 2
## [9109] 3 2 2 1 1 3 3 4 2 3 3 2 2 2 3 3 3 3 4 4 2 4 4 4 4 1 2 4 4 3 1 3 2 4 1 1
## [9145] 4 3 3 1 2 2 2 2 3 1 1 4 4 2 3 2 2 3 2 4 1 4 1 2 1 4 2 2 4 3 4 1 1 4 1 4
## [9181] 1 1 2 2 3 3 3 1 1 1 3 3 2 4 1 2 4 3 3 2 1 3 4 3 4 1 3 1 4 2 1 4 1 3 4 4
## [9217] 4 4 1 2 3 3 3 1 3 2 3 4 1 1 1 3 3 4 2 1 1 4 2 2 4 3 1 1 3 4 2 3 4 1 3 4
## [9253] 2 2 2 4 3 2 3 3 3 4 3 2 1 3 1 4 2 3 3 1 1 1 1 3 3 1 4 4 2 4 2 2 1 1 1 2
## [9289] 4 1 2 4 4 3 1 2 3 2 1 3 1 3 2 2 4 2 2 4 3 4 2 3 4 3 1 4 1 2 3 3 1 3 3 4
## [9325] 1 3 3 1 4 1 4 2 4 2 3 1 1 3 2 1 1 2 3 4 4 2 1 3 2 2 3 2 2 4 3 4 2 2 4 3
## [9361] 1 2 3 3 4 4 3 3 4 4 4 2 2 3 1 2 4 3 4 4 2 4 4 1 2 2 2 3 1 3 4 1 4 4 1 4
## [9397] 3 3 4 3 4 1 3 2 2 1 2 2 3 3 4 3 3 3 3 3 2 2 3 3 4 1 3 4 1 2 3 3 1 3 2 4
## [9433] 2 3 2 2 1 3 4 3 4 3 3 1 4 1 2 1 3 4 3 1 3 3 3 3 3 1 4 2 4 1 4 3 1 1 4 1
## [9469] 1 2 1 4 2 1 2 3 3 3 1 3 3 1 3 2 3 3 1 2 2 3 1 3 3 3 3 2 4 2 2 1 3 4 3 1
## [9505] 1 1 3 2 4 3 2 4 3 1 1 3 2 1 2 3 2 2 4 1 3 4 2 1 3 4 1 3 4 3 2 2 3 1 3 4
## [9541] 3 2 3 1 1 1 4 4 3 3 4 3 4 1 1 4 3 2 2 3 4 4 4 1 4 4 1 3 4 2 3 1 4 3 3 1
## [9577] 4 3 1 2 3 3 1 1 2 2 3 1 4 2 1 2 3 4 1 4 2 4 3 3 4 3 1 1 4 4 3 1 3 2 4 1
## [9613] 4 4 2 3 2 1 3 4 1 2 2 3 3 1 1 2 3 2 3 1 1 4 2 4 2 1 4 1 4 1 3 1 3 3 1 4
## [9649] 1 3 3 3 4 1 3 3 1 1 2 3 4 3 1 2 2 3 1 3 3 1 3 1 3 4 3 4 4 3 4 2 1 3 1 3
## [9685] 3 4 3 2 1 1 3 2 4 1 4 1 3 1 1 1 1 1 3 1 4 3 4 3 2 3 3 3 4 3 1 3 1 3 1 4
## [9721] 4 1 1 1 3 4 1 3 3 1 1 1 3 4 2 2 2 2 4 3 3 2 3 3 3 3 3 1 3 4 3 1 3 1 3 1
## [9757] 2 3 3 2 3 2 4 1 3 2 3 1 1 2 4 4 3 1 3 4 4 1 2 1 2 4 2 2 1 1 4 1 3 2 1 3
## [9793] 2 3 3 2 1 4 4 2 1 1 3 3 1 2 3 4 3 1 4 2 3 4 3 3 3 1 4 4 2 1 3 1 3 2 2 3
## [9829] 3 2 1 3 1 2 1 4 1 3 1 2 3 3 4 4 1 3 3 1 1 3 2 4 3 2 1 3 3 1 3 4 1 1 3 3
## [9865] 3 4 4 2 2 3 3 4 1 1 2 3 1 1 1 3 1 2 1 2 1 3 3 4 2 3 4 3 1 3 1 4 2 2 2 2
## [9901] 1 1 2 1 2 2 4 2 3 4 2 4 2 1 3 4 4 3 1 4 4 2 3 1 1 2 3 2 1 1 1 3 3 2 4 3
## [9937] 1 1 2 4 3 4 4 3 3 1 4 2 1 4 3 2 4 3 3 2 1 4 3 1 3 3 4 4 2 4 4 3 4 4 1 4
## [9973] 1 1 1 1 4 2 3 3 4 4 1 4 4 1 2 4 3 2 3 3 3 4 4 3 2 1 2 1 3 1 4 3 4 4 4 2
## [10009] 4 1 4 3 4 2 4 3 2 1 1 4 3 2 4 3 4 3 1 3 1 2 4 4 4 1 2 1 3 1 3 3 1 3 2 3
## [10045] 3 1 4 4 2 1 1 2 3 2 1 1 2 2 3 3 1 3 1 3 1 2 4 4 1 4 4 1 3 3 4 3 3 3 2 4
## [10081] 4 1 2 1 4 3 2 1 4 3 3 1 1 3 1 4 4 3 3 4 3 4 3 4 2 2 3 4 3 3 2 3 3 2 2 4
## [10117] 2 4 3 2 3 1 4 2 3 3 1 4 1 2 4 1 4 2 3 2 4 2 3 2 3 3 1 2 2 4 1 2 3 2 4 3
## [10153] 3 1 1 3 2 4 2 4 4 2 1 4 2 2 3 4 2 2 3 3 3 3 3 4 3 2 2 1 4 1 2 3 2 4 4 4
## [10189] 3 4 3 3 1 2 2 4 2 4 2 3 4 2 1 1 1 2 1 4 3 1 3 4 2 4 2 4 1 2 4 3 4 4 4 4
## [10225] 3 3 2 3 3 2 2 1 3 3 3 3 3 1 2 3 1 4 4 3 4 1 3 1 2 3 4 3 1 3 2 3 4 2 3 3
## [10261] 4 3 1 1 3 3 3 2 3 3 3 4 4 2 1 2 2 3 1 1 3 4 3 1 4 1 2 4 4 4 1 1 4 2 2 1
## [10297] 3 4 3 3 1 3 4 3 3 2 2 3 1 2 2 4 1 3 1 3 1 4 1 4 4 3 4 4 4 2 1 4 2 3 1 3
## [10333] 1 1 2 1 3 1 4 4 2 3 3 2 3 2 3 1 2 3 1 4 1 3 3 1 4 2 3 1 2 1 1 2 3 2 3 2
## [10369] 1 2 4 4 1 3 3 3 4 3 1 3 3 4 1 1 2 4 3 1 1 3 1 3 2 2 3 1 3 1 1 3 4 3 1 2
## [10405] 2 4 1 4 3 4 2 2 4 3 1 4 3 1 4 2 1 4 3 2 4 3 3 3 2 1 3 4 4 4 4 3 3 2 2 1
## [10441] 2 3 3 2 4 2 4 4 4 3 1 1 3 3 2 1 1 4 4 2 3 2 2 2 2 1 3 2 4 1 3 3 4 1 3 1
## [10477] 3 3 1 1 3 1 2 1 1 1 2 4 2 4 3 4 1 3 1 3 2 4 3 2 4 3 1 2 3 1 1 2 2 1 4 4
## [10513] 2 3 1 4 2 1 3 1 4 4 2 4 1 2 4 1 4 3 2 2 1 1 4 1 1 4 2 2 1 1 2 1 4 4 3 3
## [10549] 3 3 1 1 3 1 2 3 3 3 1 2 4 1 3 2 3 3 2 4 3 4 1 4 2 3 1 4 4 3 4 4 3 3 1 2
## [10585] 4 2 2 3 2 2 3 2 1 3 4 4 2 4 3 4 1 2 4 4 1 4 1 3 1 3 4 1 1 1 1 4 4 2 4 1
## [10621] 1 4 4 2 2 1 2 4 2 3 2 1 3 4 4 3 4 3 4 4 3 1 2 1 2 4 3 4 1 1 1 4 1 4 1 4
## [10657] 3 4 2 1 4 4 3 1 4 2 3 1 3 1 2 2 3 2 3 1 2 4 2 3 1 3 3 2 3 2 4 3 1 4 3 1
## [10693] 1 3 3 4 2 1 1 4 4 1 4 3 3 3 1 4 3 3 3 3 3 3 2 3 4 4 4 2 2 4 4 2 4 2 2 3
## [10729] 3 4 1 3 2 4 3 1 4 3 2 2 3 3 1 4 2 3 1 1 3 4 3 4 1 4 1 2 4 1 3 3 3 3 3 4
## [10765] 3 3 3 1 2 1 2 4 2 4 1 4 4 1 1 3 3 1 1 4 4 4 4 3 1 1 4 2 2 3 3 4 4 3 1 4
## [10801] 4 3 4 1 4 3 2 1 1 3 1 3 4 4 1 3 3 3 3 1 4 2 1 4 2 3 4 2 3 1 2 4 3 2 2 2
## [10837] 3 1 1 3 4 4 3 2 3 1 1 2 4 1 3 2 1 1 3 2 1 2 3 2 3 2 3 2 2 1 2 2 3 3 1 3
## [10873] 2 3 4 4 3 1 2 4 2 4 4 4 1 2 2 4 1 1 3 4 1 4 2 4 3 3 2 3 2 4 2 3 3 3 2 4
## [10909] 2 2 4 4 2 1 3 1 1 3 2 1 4 2 4 2 4 2 3 2 2 3 1 3 4 3 3 4 3 1 3 2 3 2 1 2
## [10945] 2 1 2 4 1 4 3 1 3 4 4 4 4 2 2 4 2 3 2 3 2 4 3 1 4 1 3 2 2 3 1 3 4 3 3 2
## [10981] 1 4 3 3 3 4 1 1 3 4 1 1 4 1 2 4 3 3 2 4 4 3 1 3 3 3 3 1 4 2 3 2 1 3 2 4
## [11017] 1 4 2 2 3 1 3 2 3 3 4 4 1 2 1 1 1 1 4 4 3 4 1 4 1 3 2 2 1 1 3 1 2 3 2 1
## [11053] 2 3 4 4 3 3 1 2 4 2 1 4 3 3 2 3 1 4 1 1 3 2 4 4 1 3 2 3 4 1 4 3 1 3 3 1
## [11089] 2 4 1 1 3 2 4 3 2 4 3 1 4 3 4 2 1 2 2 4 2 4 4 3 3 3 4 1 3 2 3 4 3 2 4 3
## [11125] 1 2 2 4 3 3 1 4 3 2 4 3 1 2 3 3 3 1 2 2 4 3 1 3 4 4 2 1 2 4 1 2 3 4 1 4
## [11161] 4 3 2 4 4 4 2 4 2 4 3 1 3 3 2 4 4 1 2 2 2 2 3 4 1 1 4 4 2 4 4 3 4 1 1 4
## [11197] 2 1 1 2 3 2 1 4 2 1 1 2 3 2 1 3 3 3 2 4 1 2 1 1 4 1 2 1 4 1 3 1 1 3 1 1
## [11233] 2 4 3 1 1 4 4 2 2 3 3 3 2 4 1 4 2 4 3 3 4 2 1 2 3 1 1 3 3 3 4 4 1 2 3 3
## [11269] 4 1 1 3 1 1 1 4 4 3 1 3 1 3 2 1 1 2 4 1 1 3 3 3 2 2 2 2 3 1 1 1 2 3 1 2
## [11305] 3 2 2 4 4 2 2 4 1 2 1 3 1 1 2 3 2 1 2 3 3 4 3 3 4 3 1 1 3 3 3 1 4 1 2 3
## [11341] 3 3 1 3 2 4 2 3 4 2 3 1 4 3 4 1 1 3 3 2 1 1 4 4 1 2 4 3 1 1 3 3 1 4 1 1
## [11377] 1 1 4 1 1 3 4 3 4 3 2 2 4 2 1 2 3 3 3 1 3 4 1 4 2 1 2 3 1 3 3 1 1 3 2 3
## [11413] 2 1 3 1 3 3 1 3 3 3 2 3 4 2 3 3 2 4 1 4 2 4 1 3 4 1 2 3 1 4 3 4 2 3 3 3
## [11449] 1 4 3 3 1 2 4 3 4 4 2 2 3 1 1 2 2 3 3 2 1 2 3 3 4 2 1 1 3 4 3 1 2 2 3 2
## [11485] 4 2 2 4 2 2 3 4 3 4 1 1 3 1 2 1 4 4 3 3 1 3 3 1 3 3 4 1 3 1 1 3 3 2 4 2
## [11521] 1 1 3 1 1 3 2 4 3 4 2 2 1 1 3 4 2 3 4 1 1 4 3 2 1 3 1 4 4 3 2 2 4 3 1 1
## [11557] 4 3 3 4 2 3 2 2 4 4 1 3 2 4 3 3 4 2 1 3 3 3 3 1 3 3 1 4 4 4 2 4 1 1 3 3
## [11593] 3 3 3 3 1 4 4 2 3 2 1 2 2 4 4 3 4 4 1 3 3 1 3 3 1 1 4 1 1 2 1 3 1 2 2 4
## [11629] 3 1 4 4 1 2 4 1 3 3 2 3 3 4 3 1 1 1 3 3 2 1 4 3 2 3 3 3 1 2 2 3 1 3 1 1
## [11665] 3 3 4 3 2 3 4 4 2 3 2 1 2 3 1 3 4 1 3 1 2 1 1 1 4 1 1 3 4 2 3 3 1 1 4 3
## [11701] 4 4 2 3 2 4 1 3 2 3 4 1 3 3 4 1 1 3 2 1 3 2 2 1 2 1 1 1 2 3 4 4 1 1 3 4
## [11737] 1 1 4 4 2 1 1 1 4 3 1 4 1 1 2 1 3 4 1 3 1 1 3 2 4 1 1 1 3 3 2 2 1 3 4 2
## [11773] 2 1 1 1 3 2 4 1 2 3 3 4 2 1 3 3 4 1 2 3 1 1 2 3 3 2 1 2 2 2 3 3 2 2 4 4
## [11809] 2 4 3 2 2 4 2 2 1 3 2 1 4 3 1 4 4 1 3 3 2 4 3 4 2 1 2 4 1 2 3 3 3 2 4 2
## [11845] 2 3 1 2 1 2 3 2 4 3 3 2 1 4 1 3 3 3 3 1 4 1 1 2 3 1 3 1 4 3 4 3 1 2 3 1
## [11881] 2 2 4 4 4 4 2 1 3 4 1 1 3 4 1 3 1 2 4 1 3 4 2 1 1 4 3 3 3 3 3 4 3 3 1 1
## [11917] 2 2 1 2 4 3 3 1 1 2 3 1 1 1 4 1 1 1 4 3 3 4 3 4 3 1 2 4 4 4 1 4 4 3 2 3
## [11953] 2 3 3 2 2 1 2 1 4 2 4 2 2 4 2 2 4 2 3 4 4 3 2 3 3 1 2 4 3 1 3 1 4 3 1 4
## [11989] 1 4 2 3 1 4 3 2 1 3 1 3 1 2 1 1 3 2 3 3 4 1 4 2 1 4 3 2 2 2 2 3 3 4 4 2
## [12025] 4 3 3 4 3 3 2 4 2 2 3 1 2 1 3 3 3 1 2 3 3 1 1 3 1 3 1 1 4 4 3 2 1 3 3 1
## [12061] 3 3 4 1 1 1 4 4 4 3 3 3 4 2 2 4 1 3 2 1 3 3 1 1 4 2 3 1 1 4 4 3 2 2 4 3
## [12097] 4 3 3 3 1 4 2 2 1 3 3 2 4 1 4 1 2 1 3 4 2 4 3 3 4 1 3 2 2 1 3 1 4 2 1 4
## [12133] 3 2 4 3 4 3 4 3 1 3 4 2 4 1 3 4 3 1 2 3 1 4 3 2 1 1 1 3 4 3 1 2 4 1 2 3
## [12169] 1 2 4 4 4 3 3 3 4 3 4 2 2 4 4 2 4 4 4 3 3 2 3 4 3 1 1 4 2 2 2 1 1 2 4 3
## [12205] 1 4 3 1 2 2 3 1 4 1 1 2 3 3 3 2 3 3 3 3 3 3 4 1 3 3 3 2 1 3 3 3 3 3 3 3
## [12241] 3 4 2 3 1 4 4 1 1 1 4 1 1 4 3 2 4 2 3 4 1 3 1 1 4 2 4 4 3 1 3 3 3 1 3 3
## [12277] 4 1 1 4 2 1 4 3 4 2 4 3 1 3 1 3 3 2 2 4 2 3 4 4 1 4 2 4 1 4 4 1 4 4 2 2
## [12313] 1 1 3 3 1 1 1 4 2 1 4 3 4 2 1 3 3 3 4 1 2 3 3 4 3 2 2 1 2 3 3 1 1 1 1 1
## [12349] 3 4 4 1 3 4 1 3 3 3 3 3 3 1 2 4 4 3 3 2 4 2 2 4 3 3 4 3 1 2 3 2 1 1 3 4
## [12385] 1 4 1 2 3 4 4 1 1 3 4 4 4 4 3 2 1 1 1 2 2 4 4 3 3 4 4 2 3 3 2 3 2 4 3 4
## [12421] 3 3 1 4 4 2 4 1 3 3 3 1 3 3 3 2 4 2 3 2 2 3 3 2 1 3 1 3 3 2 2 4 4 2 2 3
## [12457] 3 2 4 3 3 1 2 1 1 2 3 2 2 1 1 1 4 3 3 3 1 3 3 1 4 4 1 2 4 3 1 4 4 1 4 1
## [12493] 2 1 3 2 4 2 1 4 4 3 2 3 1 3 4 4 1 3 3 1 3 2 3 1 2 3 3 3 2 4 1 1 3 4 2 2
## [12529] 4 1 2 4 1 2 2 3 1 2 1 2 1 3 3 3 2 1 1 1 2 4 3 4 4 1 3 2 1 3 3 2 4 4 2 1
## [12565] 1 3 4 3 1 1 4 1 2 3 4 4 3 4 3 2 3 2 3 2 2 1 2 3 3 1 1 4 2 1 1 3 2 1 1 3
## [12601] 2 3 3 1 1 3 2 2 4 4 2 3 4 2 1 1 2 3 3 2 2 2 4 4 3 3 3 3 2 1 4 1 4 1 3 3
## [12637] 3 1 3 3 2 1 1 3 1 3 3 3 4 2 1 3 4 4 3 1 3 3 3 3 3 2 2 2 2 3 3 1 1 1 2 2
## [12673] 1 1 3 1 1 4 4 1 2 4 2 1 4 3 3 3 1 1 2 2 1 2 4 3 3 2 1 4 3 3 4 3 3 3 3 3
## [12709] 3 1 4 3 2 2 4 3 1 1 2 1 1 3 2 1 2 1 3 3 3 3 3 2 4 1 4 3 2 4 3 2 3 1 1 1
## [12745] 3 3 1 1 2 2 1 1 2 3 3 1 4 1 1 3 2 3 4 4 1 4 2 2 4 3 1 1 1 1 4 3 1 4 1 1
## [12781] 3 1 4 3 2 3 4 3 4 2 2 2 4 1 4 2 4 2 1 3 4 3 3 4 2 1 1 4 4 4 1 4 2 1 3 3
## [12817] 3 4 1 4 1 3 3 2 3 3 3 2 2 3 3 4 3 2 4 1 4 3 4 3 2 1 4 1 3 1 4 1 3 3 1 3
## [12853] 1 2 2 3 4 3 1 1 3 3 2 1 1 3 3 1 4 2 4 1 2 1 4 1 1 4 1 2 4 2 2 3 3 4 1 1
## [12889] 4 1 4 2 1 4 2 1 3 3 2 1 2 4 1 1 3 4 3 3 4 4 2 1 4 3 3 2 4 4 2 4 2 1 3 2
## [12925] 3 3 4 4 4 2 4 1 1 2 3 3 4 1 1 3 3 4 1 3 3 4 1 1 1 2 4 4 1 2 2 4 3 4 3 4
## [12961] 1 1 2 3 3 3 1 4 3 4 1 4 4 3 4 2 1 4 3 4 4 2 3 1 2 1 2 1 4 1 2 1 3 2 3 1
## [12997] 2 2 1 3 2 2 3 1 1 4 2 2 3 2 2 3 2 2 4 4 2 4 4 2 3 2 3 3 4 3 4 1 3 3 3 2
## [13033] 1 4 1 1 3 3 2 2 4 4 1 4 1 3 2 1 2 4 3 3 1 3 4 3 4 1 1 1 4 3 3 1 1 4 2 4
## [13069] 2 3 2 1 4 1 3 4 2 1 1 3 3 1 2 3 1 2 3 2 2 4 3 2 4 2 2 3 4 2 1 2 2 2 3 1
## [13105] 4 1 3 2 3 1 3 3 3 4 4 3 1 3 2 2 4 4 2 2 2 3 4 4 2 1 2 1 4 1 3 3 1 3 2 1
## [13141] 2 4 2 4 2 1 3 3 1 4 3 1 2 2 3 4 2 2 3 2 2 1 3 1 3 2 4 4 4 1 3 3 1 3 3 4
## [13177] 4 2 4 3 4 3 1 2 2 2 2 1 4 1 4 2 2 1 1 3 1 2 1 2 1 4 3 1 1 3 4 1 1 1 2 1
## [13213] 3 3 1 4 3 3 2 2 3 3 3 1 3 4 2 4 2 1 3 1 4 2 1 3 1 1 4 1 3 1 4 1 1 3 3 1
## [13249] 4 2 1 4 3 1 1 4 2 4 4 2 2 3 4 2 1 1 3 3 1 3 3 2 4 4 2 2 1 3 2 4 3 2 2 2
## [13285] 4 2 4 1 3 1 3 2 3 3 2 3 1 3 3 2 1 2 4 4 3 1 3 2 2 4 2 3 2 4 3 2 3 4 3 2
## [13321] 3 2 2 1 1 4 2 1 2 4 4 1 4 3 3 4 3 1 2 4 2 3 4 2 3 3 4 1 4 4 1 3 4 3 3 4
## [13357] 2 4 3 3 3 3 3 1 2 3 3 2 1 2 1 3 4 3 1 2 2 3 3 2 3 4 2 4 3 2 3 4 4 4 4 3
## [13393] 4 2 4 2 2 2 2 4 4 2 2 2 3 3 1 3 1 3 2 3 4 1 3 2 4 4 1 3 4 3 2 4 4 4 3 4
## [13429] 1 3 3 4 3 1 2 4 3 1 3 3 1 3 4 1 2 2 3 4 4 1 3 4 3 3 1 4 3 3 2 1 1 4 3 1
## [13465] 1 2 4 3 2 4 1 3 4 1 4 1 2 3 1 2 2 1 1 1 4 1 1 4 2 3 2 1 1 4 3 4 3 2 3 1
## [13501] 1 4 3 1 4 4 3 2 3 4 1 4 3 3 2 4 4 4 4 3 1 1 2 4 4 2 3 1 3 1 4 1 4 2 3 1
## [13537] 2 3 2 1 1 3 3 4 3 3 3 2 1 1 1 2 2 2 2 4 3 1 1 4 2 2 3 1 2 1 4 4 2 3 1 4
## [13573] 2 4 4 2 3 1 3 1 3 1 2 3 4 4 3 4 2 4 3 4 1 4 2 4 2 3 4 4 1 2 1 3 1 3 4 2
## [13609] 4 4 3 3 2 3 4 1 1 4 2 4 4 1 3 1 1 1 1 4 1 2 1 3 3 3 1 4 1 2 3 4 2 1 4 1
## [13645] 3 4 3 3 2 4 4 1 2 3 3 1 3 3 3 1 1 1 3 1 3 4 3 4 2 1 2 2 1 4 2 2 4 1 1 3
## [13681] 2 1 3 1 4 2 1 1 3 1 4 4 4 3 1 3 4 2 1 3 3 1 4 2 3 1 3 1 1 3 2 2 2 3 1 3
## [13717] 4 3 3 4 2 2 2 2 1 1 2 3 4 1 1 4 3 3 3 3 3 2 1 2 4 4 3 1 2 3 1 2 3 1 3 2
## [13753] 1 4 2 3 1 2 3 3 3 2 1 3 3 4 3 4 2 1 2 4 3 2 3 3 3 2 4 4 1 2 3 3 1 4 2 4
## [13789] 3 3 4 1 2 4 4 3 3 2 4 2 4 1 4 1 3 1 4 1 3 1 4 3 3 3 4 1 2 3 2 4 2 3 4 4
## [13825] 1 2 2 4 1 4 1 1 3 4 3 1 1 2 1 4 2 3 4 1 4 2 1 2 1 2 3 1 4 1 4 1 3 2 1 3
## [13861] 1 3 3 4 2 3 2 1 1 3 2 3 2 3 3 4 2 4 1 3 3 2 2 3 3 3 4 3 3 3 1 3 3 1 1 2
## [13897] 4 1 2 4 4 4 3 2 4 2 2 4 3 4 2 3 1 4 3 3 3 2 1 4 2 3 4 4 4 3 3 1 4 2 3 3
## [13933] 4 4 1 2 1 1 2 1 1 1 4 3 4 1 1 1 2 4 1 3 1 3 3 3 4 3 2 1 3 3 2 3 4 3 4 3
## [13969] 3 1 1 2 3 2 4 1 3 2 4 3 2 1 1 1 2 4 4 2 1 3 4 1 1 2 1 2 4 1 1 3 4 3 3 3
## [14005] 2 4 3 3 3 4 4 3 2 3 2 1 4 1 3 4 2 1 3 4 1 4 2 4 3 2 1 4 2 3 3 2 3 4 1 1
## [14041] 2 1 3 3 3 4 2 2 1 4 4 4 1 1 1 2 3 3 4 2 3 3 1 1 3 4 2 1 4 4 1 4 4 3 3 2
## [14077] 3 2 1 1 4 4 3 1 4 4 2 3 1 2 3 1 2 2 4 4 4 2 4 2 3 3 3 2 1 2 3 2 3 4 1 1
## [14113] 1 2 4 3 1 1 4 3 4 3 3 4 4 3 1 1 3 4 2 1 3 1 3 3 1 4 3 4 4 3 4 2 1 3 1 4
## [14149] 4 2 3 4 1 4 1 2 3 1 3 3 4 4 4 3 3 2 4 3 1 4 1 3 2 3 1 4 1 1 4 4 2 2 2 2
## [14185] 3 1 2 4 3 4 1 1 3 4 4 1 4 3 2 2 3 2 2 1 1 1 2 2 2 4 4 3 4 3 1 3 3 2 2 2
## [14221] 2 2 1 4 2 2 2 1 4 2 1 3 4 1 3 2 3 4 3 1 1 2 1 3 1 4 3 4 2 1 4 1 1 3 3 3
## [14257] 4 2 2 2 4 4 3 4 3 1 4 4 2 1 1 4 4 1 2 1 3 2 1 3 4 4 4 4 4 3 2 3 3 3 2 3
## [14293] 1 4 3 2 4 2 1 2 4 1 3 3 3 4 4 1 4 4 1 1 3 2 3 3 2 4 3 2 4 1 2 3 3 1 3 2
## [14329] 3 3 1 1 3 1 4 1 3 4 4 4 1 3 1 1 4 2 1 3 3 2 4 1 1 1 4 3 2 4 3 1 1 1 1 3
## [14365] 3 2 3 3 3 1 3 1 4 2 4 4 4 3 3 1 1 3 2 4 3 4 4 3 3 1 1 4 4 4 4 1 4 3 1 4
## [14401] 4 4 2 3 2 4 1 1 1 4 1 3 3 4 1 2 1 1 4 4 3 3 2 3 3 1 1 1 2 1 2 3 1 3 2 4
## [14437] 1 4 4 1 2 4 2 3 2 1 2 3 2 4 1 1 2 3 1 2 4 2 4 2 2 2 3 2 1 3 1 1 1 3 1 3
## [14473] 4 2 2 4 1 4 4 4 1 3 3 1 1 1 4 4 1 1 2 4 3 4 1 4 1 3 4 2 2 3 3 1 3 1 2 4
## [14509] 3 3 2 3 3 1 4 1 3 2 3 3 2 1 2 4 2 3 2 1 2 4 3 3 1 2 1 1 4 2 4 1 4 3 4 1
## [14545] 3 3 4 4 2 3 3 2 4 3 1 4 3 3 1 4 2 1 4 4 3 4 4 3 2 3 4 3 4 3 2 3 1 4 4 1
## [14581] 1 1 4 2 2 3 2 3 2 3 4 2 2 3 1 3 3 4 4 1 1 1 3 2 1 4 3 2 1 1 4 4 1 3 1 3
## [14617] 4 3 2 3 1 4 4 1 3 2 1 1 4 2 2 3 3 1 4 3 3 2 4 4 3 1 3 2 1 2 4 2 2 2 3 2
## [14653] 3 2 1 3 4 2 3 2 2 2 4 2 2 2 1 3 1 2 3 1 3 3 3 2 2 2 1 2 1 4 3 2 2 1 1 3
## [14689] 2 3 2 1 4 4 3 3 1 1 3 1 3 1 3 2 4 3 1 2 4 2 1 2 3 3 1 1 2 2 3 2 1 1 4 2
## [14725] 2 4 1 2 1 3 3 4 2 3 1 3 3 4 1 2 3 3 2 3 2 2 3 1 1 3 2 4 4 3 1 3 1 3 4 1
## [14761] 1 2 2 3 3 4 3 1 2 3 4 2 4 3 4 4 1 1 1 3 4 1 3 4 1 4 1 1 3 2 2 1 4 3 4 2
## [14797] 1 4 3 2 3 2 1 2 3 1 3 3 3 2 4 1 4 4 2 2 2 3 2 2 4 2 2 2 1 2 3 3 3 2 2 2
## [14833] 3 3 2 4 3 2 2 1 4 2 1 4 2 3 1 2 1 2 3 3 1 1 3 4 1 1 4 4 1 4 4 2 2 1 2 1
## [14869] 4 2 1 4 1 3 3 1 2 1 4 1 1 3 3 2 1 3 3 3 4 3 2 2 2 4 4 1 3 1 1 1 4 4 1 4
## [14905] 4 1 3 2 1 2 2 4 4 3 2 2 3 4 3 1 3 4 1 2 1 4 3 3 4 4 2 1 2 3 2 2 2 3 1 2
## [14941] 1 2 1 1 1 2 4 2 3 4 1 3 3 1 1 4 2 2 1 2 4 1 3 4 1 3 3 3 4 2 1 1 3 1 3 3
## [14977] 3 2 3 4 4 1 4 3 2 3 3 2 3 3 3 1 2 3 2 4 2 2 3 2 1 2 3 1 2 1 1 1 1 3 3 2
## [15013] 2 2 4 4 4 4 2 3 2 3 4 2 3 4 2 1 1 3 2 2 2 3 1 1 3 1 2 3 3 4 1 3 2 1 3 1
## [15049] 2 4 2 2 2 1 4 1 2 3 3 2 4 3 3 3 2 4 3 3 3 3 3 1 1 1 4 1 4 2 3 3 2 4 3 3
## [15085] 4 3 1 3 3 1 4 1 3 1 3 2 3 1 1 1 4 2 1 3 4 3 1 3 4 2 1 2 1 4 4 2 4 2 3 2
## [15121] 4 3 2 1 1 3 4 1 1 2 3 4 3 4 4 3 4 3 2 3 2 3 3 4 4 3 1 4 3 1 3 1 1 3 3 4
## [15157] 1 3 2 3 2 3 2 3 4 3 4 1 4 4 1 2 2 2 4 2 3 3 3 1 1 3 4 3 2 4 1 1 3 2 1 4
## [15193] 3 1 2 4 2 3 4 3 2 3 4 1 4 2 3 3 4 4 2 1 4 3 2 1 1 3 1 2 2 4 2 3 2 4 3 4
## [15229] 2 1 1 2 4 4 3 2 3 3 2 4 1 3 2 1 2 3 3 1 4 1 4 2 4 3 4 4 2 3 3 2 3 3 3 1
## [15265] 1 3 3 1 4 2 4 1 2 1 1 1 2 4 4 4 4 3 3 2 3 2 2 3 3 1 3 3 3 2 3 4 1 3 1 4
## [15301] 1 4 4 2 3 1 2 1 1 1 1 4 2 2 4 3 4 3 1 4 2 2 2 3 1 4 1 1 2 4 3 3 3 3 1 1
## [15337] 2 1 1 2 4 4 1 3 3 2 3 4 1 3 2 2 2 2 4 4 3 3 4 1 3 3 4 2 3 1 3 2 2 3 2 3
## [15373] 2 2 1 3 1 1 3 4 1 4 1 2 4 1 1 3 4 4 4 1 3 1 2 3 3 3 1 1 4 4 1 3 2 3 4 2
## [15409] 4 1 3 4 3 2 1 1 2 1 4 1 3 1 2 1 3 3 1 3 1 4 3 3 4 1 4 3 1 4 1 3 1 3 1 1
## [15445] 1 2 3 2 4 4 4 1 1 2 1 4 2 2 1 2 1 1 3 1 4 1 3 3 1 3 3 1 2 4 2 4 3 3 3 4
## [15481] 3 2 1 3 3 4 2 4 2 2 2 3 3 2 4 4 2 3 1 4 4 2 2 3 3 1 4 1 2 1 3 2 2 2 4 1
## [15517] 2 2 1 2 1 4 3 2 4 1 4 4 4 2 4 1 2 1 3 1 2 3 1 3 2 1 2 1 2 3 3 3 1 2 2 2
## [15553] 1 4 4 2 3 4 3 4 3 2 3 3 1 1 1 3 1 3 2 4 4 3 3 4 3 1 3 4 1 1 4 1 3 1 4 4
## [15589] 1 4 2 2 1 2 2 1 4 3 3 4 1 1 3 1 4 2 3 3 3 1 4 2 3 1 2 2 3 2 2 1 4 3 2 1
## [15625] 3 3 4 3 3 1 3 4 1 4 4 1 3 3 4 3 1 1 4 4 3 3 3 3 4 3 4 3 2 3 1 2 2 1 1 2
## [15661] 1 2 4 4 1 4 4 4 1 1 1 3 1 1 2 3 1 2 4 4 2 2 4 3 3 4 3 1 1 3 1 1 3 2 1 1
## [15697] 4 1 2 1 1 1 4 1 4 1 1 4 4 3 4 3 3 4 1 2 3 3 2 4 4 4 3 2 3 3 3 3 2 3 2 2
## [15733] 4 1 3 1 1 3 1 3 3 2 1 4 4 1 3 4 1 3 1 4 4 2 3 3 4 1 4 2 3 1 1 3 1 4 2 1
## [15769] 1 3 1 4 3 3 3 2 2 4 2 4 4 1 4 2 1 2 3 2 1 3 3 1 3 4 3 1 2 3 3 4 3 3 2 2
## [15805] 3 1 1 1 1 1 1 4 1 1 4 1 4 3 3 2 2 3 4 4 3 1 4 2 1 3 4 1 3 1 4 1 2 2 2 2
## [15841] 3 2 3 2 1 4 2 2 3 4 3 4 3 3 3 4 2 3 1 4 2 3 2 1 3 3 4 4 1 4 1 1 1 4 2 2
## [15877] 1 2 1 4 1 3 1 1 1 3 3 1 2 4 1 1 1 2 3 4 4 4 4 1 4 3 3 4 2 1 3 4 2 3 4 3
## [15913] 1 3 1 4 2 3 2 4 4 1 1 1 2 1 3 1 2 3 4 3 2 3 2 1 3 2 4 3 2 3 3 4 4 4 3 3
## [15949] 4 3 1 3 3 4 2 3 3 1 1 3 4 3 1 4 1 2 4 2 3 3 1 3 1 4 1 2 1 1 4 1 3 3 3 3
## [15985] 3 1 4 1 4 1 3 2 1 3 4 3 3 3 1 2 4 2 1 1 3 1 3 1 4 3 4 3 1 4 3 3 1 1 3 3
## [16021] 2 1 3 1 3 1 4 1 4 3 2 1 1 4 3 3 1 3 3 2 1 3 1 1 3 2 1 3 2 3 1 2 3 1 1 3
## [16057] 4 2 2 3 4 2 1 3 3 1 1 3 4 2 3 4 1 4 1 3 4 2 1 3 4 3 4 2 3 3 3 1 1 2 4 3
## [16093] 2 1 3 1 3 3 2 2 3 3 4 3 2 3 2 4 1 3 3 3 2 2 3 3 3 1 4 4 1 4 4 1 2 3 3 2
## [16129] 1 4 1 2 1 3 2 4 1 1 2 3 3 2 2 4 3 1 4 3 3 4 3 4 3 1 2 3 1 1 1 4 2 2 1 4
## [16165] 3 3 1 1 1 1 3 4 1 1 2 2 4 3 1 1 4 1 2 4 3 2 3 4 2 4 3 1 4 4 3 3 3 1 2 1
## [16201] 2 1 3 1 3 3 1 2 1 2 3 2 4 3 2 4 1 1 2 2 1 3 4 1 3 3 3 3 4 2 1 1 1 2 3 4
## [16237] 4 2 4 1 1 3 1 1 2 4 1 3 4 2 2 1 1 3 3 4 4 1 1 4 1 2 1 4 1 3 2 2 4 1 4 3
## [16273] 3 1 2 4 3 2 3 2 4 1 3 4 3 1 1 2 1 4 1 3 4 4 4 3 4 3 3 4 2 1 1 3 3 1 1 1
## [16309] 2 1 4 2 1 2 1 3 1 1 2 1 1 1 2 4 3 3 4 4 1 4 1 4 4 3 4 2 4 2 3 1 3 3 4 4
## [16345] 2 1 4 4 4 4 3 1 4 1 2 2 4 1 3 3 2 2 3 1 2 2 1 3 4 4 2 3 2 1 1 3 3 2 3 2
## [16381] 2 4 2 1 1 3 2 1 3 1 3 3 2 2 1 3 3 3 3 2 1 3 4 3 3 1 2 4 1 3 3 3 2 1 2 2
## [16417] 3 3 4 3 1 2 2 3 3 2 2 4 4 1 2 3 4 2 2 2 1 4 2 4 4 1 3 4 3 2 3 2 1 3 3 4
## [16453] 3 3 4 2 1 4 1 2 1 3 4 3 1 3 4 3 1 3 2 2 3 1 1 2 2 3 3 1 4 1 4 2 2 2 4 1
## [16489] 4 3 1 2 2 2 3 4 2 4 1 3 4 2 1 4 2 1 3 3 1 4 3 1 4 3 2 1 3 2 1 3 2 3 3 4
## [16525] 4 2 1 1 2 4 3 2 2 4 2 4 2 2 1 3 3 4 3 3 4 1 3 2 1 1 1 3 4 1 4 1 3 3 1 2
## [16561] 3 1 3 1 3 3 3 4 2 4 3 4 4 4 4 4 3 1 1 4 3 3 4 3 2 2 3 1 4 4 2 4 4 1 2 3
## [16597] 2 2 4 2 3 4 3 3 4 4 3 3 4 3 2 3 4 1 4 4 3 4 3 3 1 2 1 1 4 3 1 1 2 2 3 4
## [16633] 1 1 4 3 3 3 4 2 1 1 3 4 3 4 1 3 1 2 3 3 1 3 4 3 3 3 3 2 1 1 1 2 2 4 2 2
## [16669] 4 2 1 3 3 3 4 1 3 2 1 3 3 4 4 2 4 1 1 3 3 3 4 4 2 3 2 1 3 4 3 3 4 3 2 4
## [16705] 2 3 1 3 2 3 1 4 2 4 2 1 3 4 1 1 2 2 3 4 1 3 3 4 2 4 4 3 1 3 4 1 3 3 4 1
## [16741] 2 2 2 2 2 4 4 1 1 1 2 2 3 1 4 1 1 3 3 3 3 1 1 1 3 3 1 2 1 2 2 3 1 3 2 1
## [16777] 1 4 1 1 2 1 2 1 1 2 4 3 1 2 1 4 4 4 3 1 1 1 4 1 2 3 2 4 1 3 3 3 3 1 3 1
## [16813] 3 4 1 2 1 1 2 3 4 1 2 3 1 1 3 1 4 4 4 2 1 4 1 1 1 2 2 3 3 4 4 1 2 2 2 4
## [16849] 1 1 1 3 3 3 4 4 4 3 4 4 1 4 3 3 4 4 3 1 3 1 1 4 1 4 1 4 3 1 1 4 3 3 2 3
## [16885] 4 4 3 2 4 3 3 3 3 4 1 1 4 4 4 4 2 4 2 4 3 1 1 2 3 4 3 1 1 4 4 3 3 4 1 4
## [16921] 1 1 1 1 1 2 4 4 1 4 2 2 1 4 4 3 4 2 1 1 1 1 2 3 3 3 1 4 4 2 2 3 1 3 3 3
## [16957] 4 1 2 4 2 2 4 4 3 3 4 4 2 3 1 1 3 2 4 3 4 3 3 2 2 2 3 1 2 1 1 4 2 3 1 2
## [16993] 4 2 4 2 2 4 3 3 1 4 3 4 3 3 3 3 4 3 2 4 3 4 4 3 3 3 1 1 3 1 4 3 3 4 2 3
## [17029] 2 1 3 4 3 1 2 4 1 2 3 1 1 3 3 3 2 1 4 2 1 4 4 2 4 3 1 1 2 2 2 3 4 3 3 1
## [17065] 2 4 1 1 1 3 3 2 3 4 1 3 2 2 1 4 2 1 3 1 3 1 2 3 4 2 1 2 3 2 4 1 3 4 3 3
## [17101] 4 1 4 4 3 3 4 3 2 4 3 1 1 1 3 2 3 1 2 4 4 4 1 1 1 2 3 1 1 2 2 2 1 1 1 3
## [17137] 4 3 2 2 3 1 3 3 4 3 4 4 2 4 2 1 3 1 1 1 3 2 1 1 2 4 3 3 2 3 3 3 4 3 2 1
## [17173] 1 1 3 2 2 3 3 2 2 4 1 4 2 2 2 1 3 2 3 3 2 4 2 1 2 1 1 1 4 1 1 1 2 3 3 3
## [17209] 3 2 4 1 2 4 3 2 1 4 3 2 3 4 1 3 2 1 1 2 2 4 1 3 2 2 4 3 3 1 2 1 3 3 3 2
## [17245] 1 2 4 1 3 2 1 2 4 1 4 2 4 2 4 3 4 1 2 2 3 1 4 3 2 3 1 3 2 1 3 3 3 1 1 3
## [17281] 2 2 1 1 3 4 2 4 4 1 4 3 2 3 4 3 1 4 2 4 3 2 2 3 2 4 3 4 4 1 1 1 1 3 2 3
## [17317] 2 1 1 3 3 3 4 3 2 3 3 3 1 1 1 3 2 1 3 3 4 4 3 2 2 2 2 4 1 4 1 3 3 1 4 2
## [17353] 4 1 1 2 1 4 2 1 3 3 1 4 4 4 4 2 4 3 2 4 3 2 3 3 2 1 2 2 1 2 2 1 3 4 2 1
## [17389] 4 3 4 1 3 3 1 3 1 3 3 1 3 4 4 3 2 2 3 4 4 3 2 1 2 2 1 3 3 2 1 1 1 2 2 3
## [17425] 2 3 1 3 4 3 1 3 4 1 4 4 4 3 2 4 3 1 4 3 3 2 3 2 1 3 2 4 1 4 4 3 1 1 3 1
## [17461] 4 3 3 3 2 4 3 3 1 1 2 1 2 1 2 3 2 2 1 1 2 1 4 1 4 4 1 2 2 1 4 1 3 1 1 4
## [17497] 1 3 3 2 3 1 4 3 2 2 1 1 1 2 2 2 4 3 3 4 3 3 4 4 3 4 3 2 2 3 2 1 3 3 2 3
## [17533] 2 3 4 4 1 4 4 2 2 3 1 1 4 1 2 1 3 1 3 4 2 2 2 2 4 4 2 4 4 3 2 2 4 4 4 1
## [17569] 4 2 3 3 2 1 1 2 3 3 4 3 4 4 4 1 2 2 3 2 1 4 3 3 3 3 4 2 1 3 3 2 2 1 1 3
## [17605] 4 4 1 1 4 4 3 1 2 3 2 4 4 3 3 3 1 1 3 2 4 1 1 2 4 3 1 1 3 1 2 3 1 3 1 4
## [17641] 2 3 1 3 2 1 3 2 4 3 4 3 3 2 4 1 1 1 3 3 3 1 3 3 1 1 2 3 4 3 1 1 3 1 4 2
## [17677] 4 2 4 1 3 1 4 4 2 1 3 4 4 2 1 2 2 1 1 3 3 4 3 2 2 2 3 1 1 2 4 3 2 3 2 2
## [17713] 3 3 3 1 2 3 3 3 4 3 3 1 4 2 1 1 2 1 1 1 4 4 3 1 4 1 2 4 1 3 3 3 4 2 3 1
## [17749] 1 4 4 3 3 3 2 1 4 2 2 2 4 3 4 2 4 4 2 3 4 1 4 2 1 4 3 2 2 3 4 1 3 2 1 2
## [17785] 2 4 1 1 1 4 3 3 1 4 4 3 2 1 2 4 3 4 1 2 3 3 1 1 4 4 2 4 2 3 3 1 4 2 2 2
## [17821] 3 4 1 1 3 4 3 3 4 4 1 1 2 3 1 4 3 4 1 4 3 1 4 3 3 3 3 1 2 2 4 4 4 2 3 3
## [17857] 2 3 4 2 2 1 4 3 4 1 4 2 4 2 3 4 1 1 4 3 1 4 2 3 4 4 3 3 2 4 1 3 1 3 4 3
## [17893] 3 4 3 1 3 1 2 4 2 4 1 1 1 3 4 3 1 1 3 4 2 4 3 4 4 1 1 3 4 4 4 2 3 3 1 1
## [17929] 2 1 3 1 3 1 4 3 2 3 3 3 1 2 4 4 1 2 1 3 2 3 3 4 1 3 3 1 3 1 2 4 3 1 3 4
## [17965] 2 4 2 1 1 4 2 3 4 1 1 1 3 1 4 4 1 3 4 3 4 1 4 2 4 4 1 2 3 4 3 1 3 2 2 2
## [18001] 1 2 4 3 1 4 1 1 3 3 4 3 3 2 3 3 3 1 1 2 4 4 4 2 4 3 4 1 2 3 2 3 3 1 2 1
## [18037] 3 4 2 1 2 4 2 3 2 2 1 2 2 4 3 3 2 1 2 2 4 3 3 3 3 1 1 3 4 1 2 3 4 2 1 1
## [18073] 1 1 3 2 4 1 4 2 3 1 3 4 2 3 4 4 2 3 4 1 1 4 3 3 2 4 2 1 4 1 3 3 2 3 1 4
## [18109] 4 1 3 4 4 2 1 3 1 3 4 3 3 2 4 1 3 3 3 4 4 1 3 3 2 3 3 4 3 4 1 3 1 1 3 2
## [18145] 1 2 1 3 2 2 1 2 1 2 1 3 3 3 4 1 2 1 3 2 1 2 3 2 4 2 3 2 2 3 2 3 1 1 1 2
## [18181] 4 4 3 3 3 2 2 1 3 1 1 2 1 4 3 1 4 4 1 2 1 4 2 3 2 4 4 2 3 3 4 1 4 4 1 1
## [18217] 2 2 4 3 2 4 1 4 3 2 3 2 2 1 3 2 3 1 1 3 1 1 4 4 2 3 4 3 1 3 3 3 3 3 3 1
## [18253] 1 2 2 2 3 2 3 2 4 2 1 1 1 1 2 3 1 2 3 2 2 2 1 3 2 3 4 2 4 1 1 4 3 4 2 2
## [18289] 1 2 3 4 2 3 1 3 4 4 4 1 4 1 3 2 4 3 3 2 3 1 2 1 3 3 1 1 1 4 3 4 1 1 2 1
## [18325] 3 2 3 2 3 4 1 2 1 3 1 1 2 3 1 4 3 3 3 3 1 3 1 4 3 1 4 1 1 3 4 4 3 4 3 2
## [18361] 2 1 2 4 2 1 2 3 2 2 2 4 3 1 1 1 1 3 2 3 2 2 3 3 4 3 4 3 3 3 3 2 3 3 3 4
## [18397] 4 3 1 4 4 2 2 3 1 2 3 3 3 4 2 1 2 2 2 4 4 2 4 1 4 1 4 4 4 1 4 3 2 3 4 3
## [18433] 1 1 3 4 3 4 1 1 4 4 3 4 3 1 1 4 2 4 1 2 2 3 4 3 1 3 1 3 3 3 2 3 3 2 1 2
## [18469] 1 3 3 2 3 4 3 3 4 2 3 4 4 3 3 2 2 4 4 2 1 3 1 3 3 3 1 3 2 1 2 3 1 1 2 1
## [18505] 1 2 3 2 4 1 4 2 2 3 3 2 3 3 2 3 2 4 1 4 2 3 3 4 1 4 3 4 1 4 4 2 1 3 3 4
## [18541] 3 4 2 2 3 1 3 3 1 3 4 3 3 1 2 4 1 4 3 3 1 3 4 3 3 2 2 3 4 1 1 4 3 3 2 4
## [18577] 2 1 1 2 4 3 3 2 3 1 3 2 2 2 1 1 2 4 4 3 3 3 3 2 1 3 1 4 4 2 3 2 1 3 4 4
## [18613] 3 1 3 2 4 1 1 4 3 1 3 3 2 2 3 3 3 4 3 2 3 3 4 2 4 1 3 2 3 1 1 3 2 2 1 2
## [18649] 2 2 3 4 2 3 3 3 3 3 2 2 3 1 3 2 1 1 1 1 2 4 4 1 3 3 4 3 2 4 1 4 3 3 3 4
## [18685] 2 2 3 3 3 3 2 3 3 3 1 3 2 3 1 3 1 2 2 1 3 2 1 1 4 2 1 3 3 2 2 2 1 1 3 2
## [18721] 2 1 2 3 4 2 4 3 3 3 3 3 3 1 2 1 4 1 4 2 3 3 4 1 3 1 4 1 1 3 1 3 1 3 4 3
## [18757] 4 1 4 2 3 1 4 1 1 1 3 1 3 2 1 4 3 4 1 4 4 2 4 3 4 4 2 4 1 2 3 3 4 3 1 3
## [18793] 1 4 2 4 2 4 4 4 2 4 2 2 2 4 3 3 2 4 3 4 4 3 3 3 4 4 3 4 2 1 4 1 4 2 1 4
## [18829] 2 2 3 4 1 1 1 3 4 1 1 2 2 3 3 1 4 2 2 3 3 3 2 2 2 2 2 3 4 3 1 3 2 3 1 4
## [18865] 4 4 3 3 4 3 2 3 1 2 2 1 3 1 1 1 2 4 1 1 3 1 3 1 4 4 3 2 1 3 3 2 2 4 4 1
## [18901] 2 1 4 1 1 2 3 2 1 2 1 2 3 2 4 1 3 2 1 4 3 4 3 2 2 1 4 1 3 1 4 1 4 4 3 4
## [18937] 4 3 1 1 4 4 4 1 3 3 2 1 4 4 1 2 1 3 3 1 1 1 3 4 1 4 2 3 3 3 1 3 2 2 4 4
## [18973] 3 2 2 4 3 3 3 1 2 1 3 3 3 1 4 4 4 1 3 1 3 3 2 4 1 3 2 2 2 3 1 3 4 2 3 3
## [19009] 4 3 4 2 2 1 4 1 4 1 3 2 3 3 3 1 4 1 3 3 4 1 3 4 3 4 3 2 3 4 3 1 3 2 4 1
## [19045] 4 1 1 4 3 2 4 2 1 1 3 4 1 2 4 3 3 3 1 4 1 3 4 2 2 1 3 1 2 3 3 3 3 4 3 4
## [19081] 1 4 4 3 2 1 3 4 2 3 2 1 3 4 2 2 3 3 1 3 2 4 3 1 3 1 4 2 4 2 1 3 2 3 4 3
## [19117] 4 3 3 2 4 4 3 3 4 4 2 3 2 3 3 3 1 3 2 1 1 3 3 3 1 2 2 1 1 3 3 4 2 1 3 4
## [19153] 2 4 3 1 4 3 4 4 3 2 2 2 2 4 2 3 4 4 1 4 1 3 3 2 3 4 4 3 3 2 1 4 2 3 3 4
## [19189] 1 4 2 3 4 2 4 1 1 4 4 3 4 1 3 1 1 4 1 4 3 2 2 4 2 3 1 3 3 1 3 3 4 4 1 1
## [19225] 4 1 4 2 2 3 2 4 2 3 4 3 2 4 4 3 2 2 1 4 2 2 4 2 3 4 1 2 1 2 3 4 1 1 3 3
## [19261] 4 3 4 3 4 3 4 1 2 1 1 2 1 2 1 3 2 2 3 3 3 4 2 2 2 2 4 4 1 4 4 3 1 1 3 1
## [19297] 2 2 4 3 4 2 3 3 2 2 1 2 4 3 1 2 1 1 3 4 2 2 4 1 3 1 4 4 2 2 3 3 3 3 4 2
## [19333] 4 2 1 4 4 2 1 2 3 4 1 3 1 1 1 3 4 2 2 2 2 3 3 2 1 1 1 3 2 2 2 4 4 3 1 1
## [19369] 3 3 2 4 3 4 3 3 3 3 1 3 2 3 1 3 3 3 4 1 3 2 2 1 1 4 1 2 3 1 1 2 3 3 1 1
## [19405] 2 2 4 2 4 2 3 4 3 2 2 1 2 4 3 1 1 1 1 4 3 2 2 2 2 3 1 3 2 3 4 3 3 1 3 4
## [19441] 1 4 3 3 2 3 2 2 4 4 1 4 3 4 1 1 3 1 2 3 1 3 1 2 4 3 4 3 1 4 1 3 3 2 1 4
## [19477] 4 3 1 4 2 2 3 3 3 2 3 1 1 4 1 3 3 3 3 3 3 1 4 3 3 2 1 4 4 2 3 1 1 4 1 4
## [19513] 4 2 4 1 3 3 1 4 3 2 2 3 2 1 1 4 2 1 1 4 3 2 4 4 3 1 3 3 1 1 4 1 4 4 3 2
## [19549] 1 2 4 1 1 2 3 1 1 4 4 2 4 3 3 2 4 3 3 2 1 4 3 1 2 1 3 4 3 3 2 3 2 2 2 1
## [19585] 3 3 2 3 4 2 1 3 2 3 1 4 3 4 3 1 1 4 3 4 3 1 3 2 4 1 1 1 3 2 1 1 4 2 3 1
## [19621] 2 4 4 1 1 3 3 2 3 4 3 4 2 1 2 1 3 1 1 3 3 3 2 1 3 2 3 4 4 1 2 2 2 2 4 1
## [19657] 4 3 3 2 1 4 3 1 2 2 2 2 3 1 4 3 3 4 1 3 1 4 2 4 4 3 3 4 3 1 4 4 2 3 1 2
## [19693] 2 4 2 2 1 3 2 4 4 4 3 2 1 3 3 2 3 3 1 4 1 4 4 3 3 1 4 1 2 2 2 1 3 1 1 3
## [19729] 3 1 4 4 2 4 4 3 1 4 3 2 4 4 1 3 4 1 1 1 4 3 1 3 2 4 1 3 1 1 4 4 2 3 1 2
## [19765] 1 1 3 4 2 1 1 4 3 2 2 3 3 3 4 1 2 4 1 3 1 3 4 3 4 2 3 4 4 2 4 1 3 1 3 3
## [19801] 2 3 1 4 2 4 1 1 3 2 1 1 1 3 2 1 1 4 3 2 1 4 2 3 1 4 4 3 1 2 3 3 4 1 4 1
## [19837] 4 2 1 1 2 1 3 2 2 1 3 3 3 1 1 1 3 1 3 1 3 3 4 4 1 3 4 3 1 4 3 2 4 1 3 3
## [19873] 1 1 1 3 3 3 3 1 3 3 4 2 2 3 3 3 1 4 4 4 3 2 2 3 1 2 3 2 3 2 2 2 3 3 4 3
## [19909] 4 2 4 2 1 1 1 1 1 1 4 1 3 2 3 1 1 1 3 4 1 2 1 4 4 4 3 1 4 1 4 1 3 4 3 1
## [19945] 1 3 3 3 2 3 2 1 1 1 3 3 3 2 3 1 3 2 1 1 2 4 2 4 4 4 2 1 3 1 3 1 1 1 2 1
## [19981] 3 1 1 3 2 3 3 3 3 3 3 1 3 4 3 3 1 4 4 3 2 1 1 1 4 4 2 4 1 4 2 3 1 3 2 2
## [20017] 1 4 3 3 3 1 1 3 3 1 1 4 3 2 2 1 3 3 4 3 2 2 2 1 4 2 3 2 4 1 2 2 1 4 4 2
## [20053] 3 1 1 3 3 1 2 1 3 1 1 4 3 4 3 1 3 4 2 1 4 3 4 4 4 1 1 3 2 2 4 3 2 3 4 4
## [20089] 4 4 4 4 4 2 1 1 1 3 2 2 3 4 3 2 2 3 2 2 4 4 1 2 2 3 1 2 2 2 3 4 1 1 3 4
## [20125] 3 4 4 3 4 2 1 3 3 3 3 2 3 2 1 3 4 1 3 3 3 3 3 4 3 1 3 3 3 4 1 1 2 2 1 1
## [20161] 2 4 1 2 2 2 2 2 2 3 4 3 1 3 1 1 1 1 3 2 3 3 1 1 1 4 3 3 1 4 3 1 2 1 4 2
## [20197] 3 3 3 2 2 1 4 3 4 1 4 4 3 2 3 4 4 3 1 3 4 3 1 3 3 2 3 2 4 3 1 4 3 3 2 1
## [20233] 2 3 2 2 1 3 2 3 2 4 4 3 4 2 4 3 4 3 2 2 4 1 2 2 4 4 3 3 3 3 1 1 2 1 2 2
## [20269] 1 2 3 4 1 4 2 1 2 3 4 3 1 3 1 2 4 4 3 4 3 3 4 2 4 2 4 1 3 2 1 3 4 1 3 3
## [20305] 4 4 3 2 3 1 1 2 1 3 3 3 1 1 1 2 2 2 4 1 3 4 1 1 3 2 1 3 3 1 1 1 3 2 2 1
## [20341] 4 2 1 1 1 3 2 2 2 1 1 1 4 4 2 3 3 1 2 1 3 2 1 2 3 3 2 3 4 3 2 3 1 3 2 2
## [20377] 2 4 4 2 1 1 1 3 1 2 2 4 1 3 3 3 2 3 2 1 3 1 4 3 4 2 4 2 1 1 2 4 3 4 3 3
## [20413] 2 4 2 4 3 1 3 3 2 2 3 2 2 3 3 3 2 3 1 4 2 3 2 2 1 3 3 1 1 4 3 2 2 3 3 1
## [20449] 4 3 4 2 4 3 3 2 2 1 1 3 3 3 1 2 4 3 1 1 3 4 2 1 1 3 3 1 3 4 3 2 3 1 3 2
## [20485] 3 1 3 3 4 1 3 1 3 3 2 4 2 3 3 2 3 1 1 4 4 4 1 2 2 3 3 4 1 4 2 3 2 3 2 2
## [20521] 3 3 4 1 2 1 3 4 1 4 2 2 2 1 3 1 3 1 3 3 3 3 4 2 3 2 1 3 4 2 3 3 4 3 2 2
## [20557] 4 3 4 3 1 1 3 3 2 3 3 4 4 4 4 2 2 2 3 1 4 3 4 1 2 3 3 1 4 1 1 2 4 4 4 1
## [20593] 1 1 1 3 4 4 3 2 3 4 4 3 2 4 4 3 1 3 3 3 3 4 4 4 3 1 1 1 3 2 4 4 2 2 4 4
## [20629] 1 2 1 3 4 4 3 2 3 4 4 2 2 3 1 3 4 2 1 3 3 3 3 4 3 3 3 1 3 4 1 2 1 1 2 2
## [20665] 1 2 3 1 1 1 1 2 1 2 1 1 4 3 2 3 3 3 2 2 3 1 1 3 3 1 2 1 2 3 3 1 1 2 1 1
## [20701] 3 3 3 3 3 4 4 1 1 1 3 1 1 4 2 3 3 4 2 1 4 1 4 1 1 4 3 2 1 1 2 3 4 3 3 2
## [20737] 2 2 1 1 1 4 1 3 4 3 3 2 3 3 2 2 4 2 1 4 4 3 4 3 3 4 1 1 4 1 3 4 2 2 1 3
## [20773] 1 4 1 3 2 4 4 4 4 3 1 1 4 4 3 4 3 3 4 3 3 3 4 4 4 2 3 2 4 3 2 1 2 2 2 4
## [20809] 4 2 3 2 3 2 4 4 4 2 3 4 2 1 2 4 4 4 4 3 1 1 3 3 1 3 3 3 1 4 3 3 4 3 2 2
## [20845] 1 3 3 2 1 1 4 2 3 3 3 2 2 1 3 1 2 3 1 2 4 1 1 2 2 1 4 4 3 2 1 1 1 3 4 4
## [20881] 3 1 2 3 2 4 4 3 3 3 1 3 2 2 1 3 1 2 4 3 2 2 2 4 1 1 3 3 1 4 2 1 3 4 4 2
## [20917] 1 2 4 2 2 4 3 1 4 2 4 3 3 2 3 1 1 3 1 3 4 3 4 3 3 1 2 2 3 1 3 1 2 4 4 2
## [20953] 4 4 3 4 4 3 1 1 4 3 1 3 4 1 4 3 3 1 3 4 1 2 1 3 2 3 4 4 3 3 1 1 3 3 4 2
## [20989] 2 4 1 3 3 1 2 1 3 1 4 2 4 4 1 4 2 1 2 3 1 1 1 3 2 2 1 4 3 1 3 3 3 4 3 1
## [21025] 3 2 1 3 2 1 1 1 2 1 3 4 3 1 2 3 2 3 4 1 1 2 4 4 3 3 2 2 4 4 4 1 4 1 3 4
## [21061] 1 2 3 2 3 2 4 4 2 4 3 4 3 2 3 3 3 1 4 2 2 2 1 3 4 2 1 4 1 2 4 2 1 3 2 1
## [21097] 4 3 3 4 3 4 2 1 1 2 3 2 3 2 3 1 3 3 3 4 1 1 2 2 4 2 3 4 4 1 4 1 1 4 3 3
## [21133] 4 3 1 3 3 1 3 3 2 3 4 1 4 2 3 2 1 3 3 3 1 3 2 4 3 1 4 1 3 3 2 3 1 3 3 1
## [21169] 3 3 1 1 2 4 1 2 3 3 1 3 4 2 1 4 4 2 1 3 1 3 4 4 3 2 3 4 1 4 3 3 1 3 2 2
## [21205] 3 2 1 3 1 1 2 2 2 2 3 3 4 2 1 1 1 1 1 3 1 2 4 1 3 1 4 4 1 1 3 3 3 4 4 4
## [21241] 2 3 2 2 1 3 1 4 3 3 3 1 3 1 3 1 1 3 2 1 2 4 4 3 4 4 1 1 3 3 2 1 1 1 1 3
## [21277] 3 1 3 1 4 1 3 1 4 3 3 1 3 3 3 3 1 2 1 1 2 1 3 1 3 4 3 3 3 2 3 3 1 2 3 2
## [21313] 1 3 1 2 1 1 1 4 2 3 4 2 1 1 3 2 4 3 1 1 4 3 1 1 1 3 1 1 3 3 2 3 1 4 1 1
## [21349] 3 3 3 4 3 2 4 3 4 3 2 2 4 2 2 2 3 2 3 2 2 2 4 4 4 2 3 2 4 2 3 4 2 4 1 3
## [21385] 1 3 3 3 4 4 4 1 2 3 2 3 2 1 3 3 2 1 4 1 4 2 1 1 3 4 2 3 2 3 3 1 2 3 3 4
## [21421] 1 1 2 4 4 4 4 4 3 2 2 4 4 4 1 1 4 4 2 2 2 4 2 1 3 3 1 4 2 1 4 4 2 3 2 3
## [21457] 4 2 3 2 4 1 4 4 4 4 4 3 4 1 1 2 1 2 1 4 1 1 1 1 1 3 3 1 3 2 2 4 4 1 3 4
## [21493] 2 4 4 3 1 1 3 2 3 4 1 1 3 1 2 3 1 3 2 2 3 4 3 4 4 3 1 2 4 3 3 4 3 1 1 1
## [21529] 1 1 4 3 3 3 3 1 4 1 3 3 2 4 2 3 1 1 1 1 1 3 2 1 3 2 4 2 3 1 2 1 1 3 1 3
## [21565] 3 4 3 1 1 2 2 4 4 4 4 3 3 2 2 3 1 3 1 3 4 3 2 4 3 3 3 4 1 3 3 1 3 1 1 2
## [21601] 3 1 4 3 1 1 1 2 4 4 4 4 2 3 2 4 3 3 4 4 2 1 1 2 1 2 4 1 2 2 1 3 2 4 2 1
## [21637] 3 2 3 4 3 3 3 3 4 2 3 4 3 4 1 1 2 3 1 3 2 1 3 3 4 3 4 1 4 2 3 2 2 2 3 3
## [21673] 4 3 3 2 1 1 1 1 4 2 4 3 4 3 2 3 2 3 1 2 3 1 3 3 1 4 1 1 2 3 3 2 1 2 2 2
## [21709] 4 3 3 1 1 1 4 4 1 1 2 4 1 2 2 3 3 3 2 1 2 1 1 3 3 2 2 1 3 1 1 1 1 2 1 3
## [21745] 2 1 4 2 2 3 3 2 4 4 4 4 3 4 4 3 3 3 2 4 4 1 1 2 1 3 1 3 1 1 3 4 3 2 3 2
## [21781] 2 1 1 1 2 2 4 2 2 2 3 3 1 1 3 3 1 3 4 4 4 2 1 4 2 1 3 2 4 2 2 3 2 2 4 4
## [21817] 1 1 4 1 1 3 3 4 3 2 1 2 2 3 3 4 1 1 1 1 4 1 1 2 1 4 1 1 3 3 2 2 3 3 4 3
## [21853] 2 1 1 2 2 4 4 2 2 3 3 1 1 3 4 4 3 4 4 4 2 2 2 2 2 4 4 4 3 3 2 1 3 1 4 3
## [21889] 3 3 1 2 4 4 4 3 1 3 3 3 3 3 1 4 1 4 4 2 1 3 2 3 1 3 4 4 3 1 2 3 3 3 1 2
## [21925] 3 1 2 1 4 1 4 2 1 3 4 1 4 3 2 3 3 2 2 1 3 2 3 3 2 2 3 2 4 3 1 4 3 2 3 3
## [21961] 3 4 4 1 3 4 2 3 3 2 1 1 4 1 3 3 3 1 1 4 3 2 2 2 4 1 2 4 4 4 1 3 1 1 3 4
## [21997] 1 3 1 2 2 2 4 3 2 3 1 3 2 1 3 1 4 1 1 2 3 1 3 4 3 3 3 3 3 2 3 1 3 3 3 4
## [22033] 3 3 2 3 2 3 3 2 2 1 4 1 3 3 3 4 4 1 3 3 2 2 1 1 2 2 4 2 4 2 1 1 2 3 4 1
## [22069] 3 4 2 2 4 1 2 2 4 4 1 1 4 2 3 1 1 1 1 1 3 4 2 1 1 4 2 1 2 3 2 3 1 1 1 2
## [22105] 3 4 4 4 1 1 1 3 1 1 4 1 4 2 4 4 3 4 3 2 4 3 2 4 2 1 1 1 2 1 1 3 3 3 3 1
## [22141] 1 4 4 3 2 1 4 1 2 3 4 2 4 2 1 2 3 1 2 3 1 3 3 1 4 2 4 1 2 1 3 3 2 4 1 4
## [22177] 4 4 1 1 1 3 1 2 3 1 2 3 3 4 4 3 2 2 3 4 1 3 1 3 2 1 4 2 2 4 3 3 1 4 1 2
## [22213] 4 1 3 3 3 1 2 1 4 3 4 3 4 4 4 3 3 2 2 1 2 4 3 1 1 2 2 1 3 2 2 1 3 2 2 2
## [22249] 2 3 3 4 4 4 4 1 2 3 3 4 4 1 3 3 1 4 4 2 2 2 2 2 1 3 1 3 2 3 1 4 2 2 2 4
## [22285] 3 3 3 3 4 3 3 3 3 3 3 1 1 1 4 4 2 3 2 1 2 3 1 2 1 3 3 2 4 3 2 1 2 3 4 3
## [22321] 1 3 1 4 4 1 3 2 3 4 3 4 1 3 1 2 1 4 4 1 1 4 4 3 3 3 4 4 1 1 2 2 3 3 2 4
## [22357] 4 2 4 2 1 2 4 4 2 3 3 3 4 1 4 4 3 4 3 1 4 3 4 4 2 3 3 2 3 1 4 1 4 3 1 2
## [22393] 3 1 1 3 2 3 1 1 3 4 3 4 4 4 1 2 4 1 3 4 1 3 4 2 2 3 1 3 1 1 2 2 1 1 1 2
## [22429] 1 4 3 2 1 3 3 4 3 2 3 4 1 2 1 1 4 1 2 3 3 4 3 3 4 3 2 1 3 3 3 1 2 1 3 4
## [22465] 4 1 3 3 2 3 1 3 3 2 4 3 4 3 4 4 3 4 3 4 1 1 3 4 3 1 3 3 4 4 1 4 3 1 2 3
## [22501] 4 4 3 2 3 3 3 3 4 3 2 4 1 3 4 3 3 3 1 2 4 2 2 4 3 3 4 1 1 3 1 3 2 1 2 4
## [22537] 3 1 1 3 3 4 1 1 3 1 4 4 4 1 2 4 2 1 3 2 3 1 3 4 4 4 4 3 4 4 2 4 1 1 4 4
## [22573] 1 1 4 3 1 4 2 1 2 3 1 3 4 3 2 1 3 3 2 2 4 2 3 3 4 4 1 1 1 3 3 4 3 4 4 4
## [22609] 3 4 3 1 1 2 3 4 1 2 3 2 3 4 1 2 4 4 3 3 3 3 4 1 4 3 2 3 1 3 1 3 1 3 1 2
## [22645] 1 4 2 3 2 4 3 1 3 2 4 4 3 3 4 2 1 3 1 4 2 3 4 4 2 3 2 3 3 3 4 4 2 2 3 1
## [22681] 3 3 3 2 2 1 1 2 2 1 4 1 2 3 4 2 1 4 4 2 2 4 1 3 3 2 4 1 4 1 4 1 4 2 1 3
## [22717] 3 2 3 3 2 4 3 2 4 1 3 2 1 3 3 1 1 2 4 4 2 4 2 3 2 1 2 3 3 2 1 2 3 4 3 3
## [22753] 2 1 2 4 1 1 3 2 1 1 1 1 3 3 2 1 1 3 4 1 4 2 2 4 2 3 2 3 2 4 3 2 1 1 4 2
## [22789] 4 4 4 4 3 3 1 2 1 3 1 2 4 2 1 4 4 3 1 4 3 3 2 4 3 3 3 2 4 2 3 3 3 1 2 2
## [22825] 3 3 2 4 3 4 4 2 2 1 1 3 2 3 4 2 1 3 2 3 1 2 3 4 3 2 3 1 3 4 2 3 2 2 3 4
## [22861] 2 4 2 1 4 2 4 4 1 2 1 4 1 4 2 3 1 2 4 3 1 4 2 3 3 3 3 1 2 1 2 2 1 2 3 1
## [22897] 2 1 3 3 4 3 3 1 4 3 3 2 1 3 3 4 3 3 1 1 1 4 2 3 2 3 1 2 2 1 2 4 1 1 1 2
## [22933] 4 3 3 2 1 4 2 4 1 3 1 4 4 2 1 4 1 1 4 4 3 4 3 3 4 2 2 1 4 1 2 2 2 3 3 1
## [22969] 4 1 4 1 3 3 3 3 2 2 1 3 4 2 2 1 3 3 4 4 2 3 3 4 3 3 4 4 2 4 2 4 1 2 3 4
## [23005] 3 2 4 2 2 3 2 2 3 1 3 2 3 2 4 3 1 2 3 2 1 3 2 2 3 4 2 1 2 3 2 2 2 4 4 3
## [23041] 2 2 3 2 3 2 2 4 2 2 3 2 1 3 1 3 1 1 3 3 2 1 4 4 1 4 3 2 1 3 1 4 2 4 2 4
## [23077] 1 3 4 4 3 4 2 3 3 4 2 3 3 3 3 2 1 1 4 2 1 3 4 2 4 4 3 3 1 4 1 4 4 4 1 1
## [23113] 1 3 2 1 2 2 4 4 1 2 3 2 2 2 4 3 3 1 4 3 1 3 4 3 2 4 4 3 3 1 1 2 3 3 2 4
## [23149] 3 3 4 1 4 4 1 4 1 4 4 1 4 1 2 2 3 4 2 4 1 3 1 3 1 1 3 1 4 4 4 2 3 4 1 2
## [23185] 1 3 4 1 1 1 1 1 3 3 4 1 4 4 2 3 4 2 3 3 1 4 2 3 3 4 3 1 2 4 1 3 2 1 1 1
## [23221] 3 3 4 3 3 3 4 4 4 1 2 1 2 1 3 1 4 4 1 2 3 1 4 3 4 1 3 4 2 3 2 1 1 4 4 4
## [23257] 3 4 1 2 4 3 2 4 3 4 3 1 1 3 4 2 4 2 1 2 3 4 2 4 4 2 1 1 1 2 1 1 3 1 1 4
## [23293] 4 3 2 4 4 2 4 1 3 2 4 3 1 1 3 1 3 3 3 1 3 4 3 2 4 3 3 4 3 2 3 3 4 2 4 4
## [23329] 4 3 1 2 3 3 3 1 2 4 4 1 4 3 3 4 1 1 2 4 4 3 4 3 4 3 3 3 2 3 1 4 1 4 3 3
## [23365] 2 4 3 1 1 1 2 4 4 3 3 3 1 3 4 1 1 2 3 1 1 3 1 2 4 3 3 2 2 4 3 3 2 1 4 4
## [23401] 4 3 1 3 3 3 1 4 4 4 3 1 4 3 2 3 2 3 4 2 4 4 3 3 3 2 4 3 1 3 3 2 3 2 4 2
## [23437] 3 3 1 3 3 4 3 1 3 1 1 2 3 1 1 2 2 2 4 3 3 3 4 3 3 2 3 2 1 4 3 3 4 3 3 3
## [23473] 2 1 1 2 1 3 1 3 1 3 2 4 1 3 1 4 2 1 1 2 3 3 4 1 3 2 1 3 2 3 1 4 3 3 3 2
## [23509] 2 4 3 4 1 1 1 4 4 3 4 1 4 1 3 4 2 1 4 2 4 4 1 2 1 1 4 3 3 1 1 2 3 3 1 3
## [23545] 3 1 3 4 3 3 1 3 4 4 2 1 3 4 3 4 4 2 3 1 1 2 1 1 1 2 2 2 1 1 3 2 2 4 4 2
## [23581] 3 3 1 3 1 4 3 1 1 2 1 3 2 2 1 2 4 1 3 3 2 2 1 2 2 1 2 3 4 3 2 1 2 4 2 3
## [23617] 4 2 1 3 2 2 4 3 2 2 3 1 2 1 1 3 2 4 1 3 1 2 3 3 3 1 4 4 3 1 4 1 4 2 4 3
## [23653] 1 3 2 4 4 1 4 2 4 3 4 2 1 2 4 2 1 4 2 3 2 1 2 3 3 1 1 4 2 3 1 3 3 3 1 3
## [23689] 1 1 3 3 4 4 3 1 1 1 2 4 3 4 2 2 2 3 3 1 1 3 3 1 4 1 3 3 2 2 2 3 3 4 1 1
## [23725] 2 3 2 2 3 2 4 1 2 3 3 3 1 3 2 3 3 4 4 3 3 1 2 3 1 4 3 1 3 4 2 2 3 4 1 4
## [23761] 4 3 1 3 3 4 3 2 1 1 3 3 3 1 3 2 2 1 2 3 1 3 2 1 4 3 4 3 1 2 2 3 3 3 1 1
## [23797] 1 2 1 1 3 1 1 2 4 3 1 2 1 1 2 2 1 1 3 3 3 1 1 3 3 2 3 1 2 1 1 2 4 3 1 4
## [23833] 3 2 1 3 2 3 1 1 3 4 3 3 4 3 3 4 1 1 1 1 3 1 3 1 4 4 3 3 1 2 1 2 3 1 4 4
## [23869] 3 3 4 1 1 1 1 3 4 4 1 1 4 2 1 3 4 1 3 4 2 4 1 1 4 4 1 2 3 4 2 4 2 2 3 3
## [23905] 3 3 2 4 1 3 4 4 1 3 1 1 4 3 2 1 3 1 4 4 2 2 4 3 1 4 3 4 1 3 2 1 3 1 3 3
## [23941] 4 3 2 2 2 1 2 2 2 2 2 3 2 1 2 2 3 1 1 1 4 3 2 4 4 2 3 3 2 1 4 1 1 3 2 1
## [23977] 3 2 3 4 4 3 1 2 1 4 1 3 3 4 1 2 2 1 3 3 4 3 4 3 4 3 3 4 2 2 2 3 3 1 3 1
## [24013] 2 1 3 4 4 2 4 1 3 1 2 4 4 2 1 4 1 3 3 4 3 1 2 3 3 1 2 2 2 2 4 3 2 2 1 4
## [24049] 3 2 3 1 1 3 4 3 3 3 2 2 1 3 3 1 4 2 1 2 4 2 3 2 2 3 4 2 4 2 1 3 1 2 2 1
## [24085] 4 4 4 4 4 3 1 3 3 1 4 4 1 3 1 1 3 2 3 1 1 2 2 4 4 1 3 3 4 4 1 4 1 2 1 3
## [24121] 1 1 4 1 4 4 4 1 4 4 3 3 1 3 3 1 2 3 4 4 2 4 3 2 3 1 4 1 3 1 3 3 3 3 3 1
## [24157] 1 4 1 1 3 2 3 4 1 2 4 4 2 3 2 3 2 2 2 2 3 3 4 1 1 1 2 1 3 1 1 1 3 1 2 3
## [24193] 3 1 4 3 1 4 2 2 4 2 1 1 4 2 2 4 3 3 3 3 2 4 4 3 4 2 4 1 2 3 2 1 3 3 2 2
## [24229] 1 3 3 4 3 1 2 3 4 3 1 3 4 3 3 2 4 4 1 2 1 2 1 2 4 1 1 3 4 3 1 4 2 1 2 2
## [24265] 3 2 3 3 1 1 4 1 2 4 2 3 2 1 3 2 2 3 3 4 4 2 3 4 4 3 4 4 4 2 3 3 1 3 3 4
## [24301] 4 3 3 2 1 3 4 3 4 4 1 3 3 1 3 3 4 2 3 3 1 2 2 3 2 4 3 3 1 4 2 1 4 3 2 2
## [24337] 3 4 2 2 1 1 4 3 1 4 2 1 1 2 2 4 4 3 1 4 3 4 4 3 1 1 1 1 3 3 2 1 3 4 4 1
## [24373] 3 4 1 1 3 1 2 2 2 3 2 3 3 3 4 3 1 2 3 2 3 4 2 2 3 1 1 4 2 2 4 3 1 4 3 2
## [24409] 4 4 4 4 1 2 3 2 3 3 1 4 2 4 3 2 3 2 3 4 4 1 3 1 3 1 2 3 1 4 1 1 2 4 1 3
## [24445] 3 4 4 1 2 2 2 3 2 4 1 2 3 1 1 2 3 3 4 2 4 1 4 4 1 3 1 1 3 1 3 3 1 1 4 3
## [24481] 1 3 3 1 2 1 2 4 4 4 3 3 3 2 3 4 3 3 2 3 3 2 4 2 4 4 4 1 3 3 4 3 3 3 4 1
## [24517] 3 3 1 4 1 3 3 2 3 3 4 1 3 4 1 2 3 3 4 4 3 1 1 1 2 4 3 4 3 2 4 2 4 1 3 4
## [24553] 1 3 1 3 1 2 4 3 4 3 2 3 2 1 3 3 3 2 3 4 3 3 1 2 3 2 1 3 2 2 1 2 1 3 1 4
## [24589] 3 1 4 3 4 1 3 1 3 3 1 1 4 1 1 4 3 3 1 4 3 3 4 3 4 1 2 4 3 1 1 2 3 2 1 2
## [24625] 3 1 3 2 2 2 2 1 3 3 4 3 3 1 4 2 1 3 1 3 2 1 4 1 3 3 4 4 4 4 2 3 3 1 4 4
## [24661] 1 4 2 3 2 2 2 2 2 3 1 3 3 3 1 4 1 1 4 3 3 4 3 4 3 1 2 4 1 2 3 1 1 3 2 1
## [24697] 1 2 1 3 1 2 4 4 2 1 1 1 4 3 2 3 4 3 2 3 4 2 1 4 1 3 4 3 4 1 1 3 2 3 3 2
## [24733] 2 1 2 4 3 3 4 1 1 4 2 3 4 1 4 4 2 4 3 1 3 2 2 1 3 2 1 4 1 3 3 3 3 3 2 1
## [24769] 1 4 4 1 1 2 3 1 1 1 2 1 3 2 4 3 1 3 4 4 2 1 2 3 2 2 1 2 4 2 2 1 2 2 3 3
## [24805] 3 2 1 3 2 3 4 4 1 3 4 4 4 4 2 3 2 4 1 4 2 4 4 1 2 2 3 2 2 3 2 3 4 2 2 1
## [24841] 4 4 2 1 4 4 3 3 1 1 4 1 4 3 1 4 1 3 4 3 2 3 3 1 3 1 1 3 4 1 1 3 4 3 4 4
## [24877] 3 4 4 1 3 3 2 1 3 4 2 2 2 2 1 3 1 3 3 4 4 1 3 4 3 3 2 1 1 4 2 1 1 3 2 4
## [24913] 4 4 1 4 1 3 1 4 1 3 3 3 3 3 2 3 2 2 2 3 2 1 3 2 1 1 1 4 1 2 2 2 4 4 3 1
## [24949] 2 4 4 4 3 1 3 2 4 2 2 2 2 2 1 1 3 1 1 3 4 4 3 4 2 4 4 2 2 1 3 3 1 4 1 3
## [24985] 2 4 2 1 4 4 3 1 2 1 2 1 1 2 2 3 1 2 3 1 1 3 2 4 1 1 3 2 2 1 4 3 1 2 1 3
## [25021] 4 2 3 1 4 2 4 4 4 1 4 4 3 3 1 2 4 4 1 2 4 3 3 1 4 3 3 2 1 3 2 1 3 2 1 3
## [25057] 1 3 2 2 3 4 3 2 4 3 3 2 3 1 2 1 1 3 1 3 4 1 1 3 4 3 1 2 3 4 2 1 3 3 2 3
## [25093] 4 2 2 2 2 3 1 1 3 2 1 2 1 3 4 2 1 3 1 2 4 1 3 4 2 3 4 4 4 3 2 2 3 3 1 4
## [25129] 1 3 4 4 4 4 2 1 3 3 1 2 1 2 2 4 3 3 1 2 4 4 3 3 3 4 3 2 2 2 1 3 3 1 2 3
## [25165] 3 1 1 2 2 2 1 2 2 1 3 4 2 3 1 4 1 2 1 1 1 1 1 4 4 2 4 2 2 3 3 3 4 3 2 3
## [25201] 3 4 3 4 3 4 1 3 2 3 1 3 3 3 3 3 4 1 1 2 4 3 1 1 3 1 4 4 1 1 2 3 3 2 2 2
## [25237] 1 3 1 1 3 3 1 2 2 1 4 1 2 3 1 2 1 4 1 3 3 4 3 3 3 3 3 3 3 2 2 4 3 3 3 4
## [25273] 2 3 4 3 1 2 3 4 1 1 3 1 2 2 4 2 1 2 3 4 3 4 1 1 2 1 3 4 2 3 3 1 3 3 4 3
## [25309] 1 1 4 3 2 3 4 2 1 3 1 1 4 4 3 2 2 4 1 2 1 4 4 4 2 2 4 4 1 4 3 1 4 4 1 3
## [25345] 3 2 4 2 4 2 4 3 3 4 3 3 1 3 2 4 3 2 1 3 3 3 2 1 3 1 2 1 3 1 2 4 1 2 1 4
## [25381] 3 4 1 3 3 2 1 2 1 1 3 4 2 4 4 4 4 2 3 2 4 3 3 2 3 2 3 1 3 2 1 3 4 3 2 4
## [25417] 1 3 2 4 4 3 4 1 2 1 3 1 3 2 3 3 1 4 3 3 2 4 3 2 2 4 2 3 4 3 3 4 1 1 3 1
## [25453] 1 2 3 3 3 1 3 3 1 3 1 4 3 4 1 1 2 1 1 2 3 3 1 2 3 1 4 1 1 4 1 3 4 3 2 3
## [25489] 4 3 3 1 4 4 4 2 3 2 3 2 3 1 1 3 2 2 2 3 4 2 2 2 2 3 3 3 3 2 4 3 1 4 3 1
## [25525] 1 3 2 3 4 3 3 1 4 3 2 1 2 1 4 3 1 3 3 1 3 1 2 2 4 2 2 3 3 2 1 1 1 3 2 1
## [25561] 4 4 3 1 3 2 2 1 4 1 3 1 4 2 3 4 1 1 2 2 1 1 4 2 3 3 4 3 3 2 4 3 1 1 3 1
## [25597] 2 4 2 4 3 3 1 1 3 4 3 2 1 4 1 4 1 3 4 3 1 1 1 3 4 2 4 1 1 2 3 4 3 4 3 3
## [25633] 1 4 4 1 3 1 3 4 3 3 3 2 1 1 4 4 2 4 2 1 2 1 4 3 3 1 4 3 4 2 4 3 3 3 1 1
## [25669] 3 3 1 4 1 4 4 3 2 2 4 3 4 3 2 1 1 3 1 1 4 3 1 1 2 3 3 2 4 1 4 2 4 3 1 4
## [25705] 3 2 3 4 1 2 4 2 2 1 4 4 2 4 4 4 3 2 3 4 3 2 1 2 1 2 3 2 1 1 2 3 4 3 4 3
## [25741] 4 3 4 2 3 4 1 4 3 2 2 2 1 2 1 3 4 2 2 2 2 3 3 3 2 1 3 2 4 3 1 3 3 3 4 1
## [25777] 1 3 1 1 1 3 2 2 4 4 3 3 3 3 4 4 3 2 1 3 1 1 1 3 1 3 3 1 2 3 3 1 1 1 1 1
## [25813] 1 3 2 1 2 1 3 3 4 2 1 4 4 3 2 2 2 3 4 2 2 1 2 1 1 3 4 2 3 4 1 1 3 1 4 3
## [25849] 2 1 1 4 3 4 3 3 3 3 1 3 1 1 4 1 3 3 4 3 4 2 3 4 4 1 1 3 4 1 3 2 4 2 4 1
## [25885] 3 4 2 3 3 2 1final_predicted = factor(final_predicted, levels=c(1,2,3,4,5))
actual_labels = factor(airline_test_set_filtered$Inflight.service, levels=c(1,2,3,4,5))
confusionMatrix(final_predicted, factor(actual_labels))## Confusion Matrix and Statistics
##
## Reference
## Prediction 1 2 3 4 5
## 1 1197 2012 1584 1350 421
## 2 192 305 2424 1647 1122
## 3 214 331 803 5255 1211
## 4 172 190 194 1100 4167
## 5 0 0 0 0 0
##
## Overall Statistics
##
## Accuracy : 0.1315
## 95% CI : (0.1274, 0.1357)
## No Information Rate : 0.3612
## P-Value [Acc > NIR] : 1
##
## Kappa : -0.0605
##
## Mcnemar's Test P-Value : <2e-16
##
## Statistics by Class:
##
## Class: 1 Class: 2 Class: 3 Class: 4 Class: 5
## Sensitivity 0.67437 0.10747 0.16044 0.11762 0.0000
## Specificity 0.77745 0.76641 0.66432 0.71443 1.0000
## Pos Pred Value 0.18236 0.05360 0.10276 0.18891 NaN
## Neg Pred Value 0.97009 0.87461 0.76755 0.58880 0.7327
## Prevalence 0.06856 0.10961 0.19331 0.36121 0.2673
## Detection Rate 0.04623 0.01178 0.03101 0.04249 0.0000
## Detection Prevalence 0.25352 0.21977 0.30180 0.22490 0.0000
## Balanced Accuracy 0.72591 0.43694 0.41238 0.41603 0.5000regularization term을 포함해서 모델을 fitting 후에, 정규화 항이 없는 모델과 비교를 한 결과, 오히려 test data set에서의 민감도에 대한 성능이 더 낮아졌다. 제일 높은 민감도값을 기준으로 0.7380 -> 0.5619(Class 4) 감소했다.
natural spline model
table(airline_train_set$Inflight.service)##
## 0 1 2 3 4 5
## 3 7063 11414 20227 37846 27041table(airline_test_set$Inflight.service)##
## 0 1 2 3 4 5
## 2 1775 2838 5005 9352 6921확인결과, 0에 대한 class의 수가 매우 부족한 것을 알 수 있다. 0에 대한 클래스 데이터를 제거하기로 한다.
airline_test_set_filtered = airline_test_set[airline_test_set$Inflight.service != 0, ]
airline_train_set_filtered = airline_train_set[airline_train_set$Inflight.service != 0, ]
table(airline_test_set_filtered$Inflight.service)##
## 1 2 3 4 5
## 1775 2838 5005 9352 6921table(airline_train_set_filtered$Inflight.service)##
## 1 2 3 4 5
## 7063 11414 20227 37846 27041종속변수의 클래스가 0인 경우가 제거된 것을 확인할 수 있다.
library(splines)
max_degree = 10
cv_errors_sp = rep(NA, max_degree)
trainX = model.matrix(~ ns(On.board.service, df = 4), data = airline_train_set_filtered)[, -1]
trainY = airline_train_set_filtered$Inflight.service
testX = model.matrix(~ ns(On.board.service, df = 4), data = airline_test_set_filtered)[, -1]fit = cv.glmnet(trainX, trainY, family= "multinomial")
best_lambda = fit$lambda.min
print(best_lambda)## [1] 0.0006734283cv를 통해서 얻은 최선의 lambda 값으로 model을 fitting 한다.
final_sp_model = glmnet(trainX, trainY, fmaily = "multinomial", lambda = best_lambda)prediction_sp = predict(final_sp_model, newx = testX, s = best_lambda, type = "class")prediction_sp = round(prediction_sp)
actual = airline_test_set_filtered$Inflight.service
prediction_sp = factor(prediction_sp, levels = c(1,2,3,4,5))
actual = factor(actual, levels = c(1,2,3,4,5))#confusion matrix
confusionMatrix(prediction_sp, factor(actual))## Confusion Matrix and Statistics
##
## Reference
## Prediction 1 2 3 4 5
## 1 0 0 0 0 0
## 2 0 0 0 0 0
## 3 1197 2012 1584 1350 421
## 4 406 636 3227 6902 2333
## 5 172 190 194 1100 4167
##
## Overall Statistics
##
## Accuracy : 0.4887
## 95% CI : (0.4826, 0.4948)
## No Information Rate : 0.3612
## P-Value [Acc > NIR] : < 2.2e-16
##
## Kappa : 0.2721
##
## Mcnemar's Test P-Value : NA
##
## Statistics by Class:
##
## Class: 1 Class: 2 Class: 3 Class: 4 Class: 5
## Sensitivity 0.00000 0.0000 0.31648 0.7380 0.6021
## Specificity 1.00000 1.0000 0.76156 0.6008 0.9127
## Pos Pred Value NaN NaN 0.24132 0.5111 0.7156
## Neg Pred Value 0.93144 0.8904 0.82299 0.8022 0.8628
## Prevalence 0.06856 0.1096 0.19331 0.3612 0.2673
## Detection Rate 0.00000 0.0000 0.06118 0.2666 0.1609
## Detection Prevalence 0.00000 0.0000 0.25352 0.5216 0.2249
## Balanced Accuracy 0.50000 0.5000 0.53902 0.6694 0.7574Conclusion
Inflight.service 예측에 대한 각 모델의 accuracy를 비교한다.
1. Linear regression model : 0.5917
2. polynomial regression model(order : 4) : 0.4887
3. Polynomial regression model with regularization(ridge) : 0.5352
4. natural spline model : 0.4887
결과적으로는 linear regression model의 정확도가 가장 높았다.
regularization term을 사용한 다항모델의 경우, 그렇지 않은 모델보다 높은
정확도를 보였지만, 해당 모델 역시 선형모델의 성능을 넘어서지는
못했다.
Tree model : satisfaction 예측
우선, 모든 변수를 이용해서 tree 모델을 생성한다.
tree_model_all = rpart(satisfaction ~ . -Flight.Distance -Departure.Delay.in.Minutes -Arrival.Delay.in.Minutes, data = airline_train_set)summary(tree_model_all)## Call:
## rpart(formula = satisfaction ~ . - Flight.Distance - Departure.Delay.in.Minutes -
## Arrival.Delay.in.Minutes, data = airline_train_set)
## n= 103594
##
## CP nsplit rel error xerror xstd
## 1 0.33564556 0 1.0000000 1.0000242 0.0008352578
## 2 0.12871693 1 0.6643544 0.6643802 0.0029027750
## 3 0.05427203 2 0.5356375 0.5356761 0.0031438054
## 4 0.02957519 4 0.4270934 0.4271468 0.0029321065
## 5 0.02820567 5 0.3975183 0.3975665 0.0029275042
## 6 0.01904935 6 0.3693126 0.3693661 0.0027334211
## 7 0.01490694 7 0.3502632 0.3503195 0.0027208857
## 8 0.01055337 10 0.3055424 0.3056090 0.0024667627
## 9 0.01000000 11 0.2949890 0.2959254 0.0024837119
##
## Variable importance
## Online.boarding Inflight.wifi.service Seat.comfort
## 24 19 11
## Inflight.entertainment Ease.of.Online.booking Class
## 10 10 9
## Type.of.Travel Cleanliness Customer.Type
## 9 1 1
## On.board.service Age Checkin.service
## 1 1 1
## Food.and.drink Inflight.service Leg.room.service
## 1 1 1
##
## Node number 1: 103594 observations, complexity param=0.3356456
## mean=0.4333938, MSE=0.2455636
## left son=2 (52271 obs) right son=3 (51323 obs)
## Primary splits:
## Online.boarding < 3.5 to the left, improve=0.3356456, (0 missing)
## Class splits as RLL, improve=0.2539593, (0 missing)
## Type.of.Travel splits as RL, improve=0.2015964, (0 missing)
## Inflight.entertainment < 3.5 to the left, improve=0.1726111, (0 missing)
## Inflight.wifi.service < 3.5 to the left, improve=0.1676912, (0 missing)
## Surrogate splits:
## Seat.comfort < 3.5 to the left, agree=0.730, adj=0.455, (0 split)
## Inflight.wifi.service < 3.5 to the left, agree=0.714, adj=0.424, (0 split)
## Ease.of.Online.booking < 3.5 to the left, agree=0.676, adj=0.346, (0 split)
## Class splits as RLL, agree=0.670, adj=0.334, (0 split)
## Inflight.entertainment < 3.5 to the left, agree=0.659, adj=0.311, (0 split)
##
## Node number 2: 52271 observations, complexity param=0.05427203
## mean=0.1489162, MSE=0.1267402
## left son=4 (50507 obs) right son=5 (1764 obs)
## Primary splits:
## Inflight.wifi.service < 0.5 to the right, improve=0.19748610, (0 missing)
## Class splits as RLL, improve=0.09211006, (0 missing)
## Ease.of.Online.booking < 0.5 to the right, improve=0.08452210, (0 missing)
## Online.boarding < 0.5 to the right, improve=0.06366470, (0 missing)
## Type.of.Travel splits as RL, improve=0.06305692, (0 missing)
## Surrogate splits:
## Ease.of.Online.booking < 0.5 to the right, agree=0.971, adj=0.151, (0 split)
## Online.boarding < 0.5 to the right, agree=0.968, adj=0.041, (0 split)
## Cleanliness < 0.5 to the right, agree=0.966, adj=0.003, (0 split)
## Inflight.entertainment < 0.5 to the right, agree=0.966, adj=0.002, (0 split)
##
## Node number 3: 51323 observations, complexity param=0.1287169
## mean=0.7231261, MSE=0.2002148
## left son=6 (10451 obs) right son=7 (40872 obs)
## Primary splits:
## Type.of.Travel splits as RL, improve=0.3186590, (0 missing)
## Class splits as RLL, improve=0.2299112, (0 missing)
## Inflight.entertainment < 3.5 to the left, improve=0.1482971, (0 missing)
## Leg.room.service < 3.5 to the left, improve=0.1452057, (0 missing)
## On.board.service < 3.5 to the left, improve=0.0990183, (0 missing)
## Surrogate splits:
## Class splits as RLR, agree=0.822, adj=0.127, (0 split)
## Age < 60.5 to the right, agree=0.814, adj=0.085, (0 split)
## Leg.room.service < 0.5 to the left, agree=0.801, adj=0.022, (0 split)
##
## Node number 4: 50507 observations, complexity param=0.05427203
## mean=0.1193498, MSE=0.1051054
## left son=8 (45789 obs) right son=9 (4718 obs)
## Primary splits:
## Inflight.wifi.service < 3.5 to the left, improve=0.27369580, (0 missing)
## Class splits as RLL, improve=0.10150640, (0 missing)
## Ease.of.Online.booking < 3.5 to the left, improve=0.07538054, (0 missing)
## Type.of.Travel splits as RL, improve=0.06892632, (0 missing)
## Leg.room.service < 3.5 to the left, improve=0.06340985, (0 missing)
##
## Node number 5: 1764 observations
## mean=0.9954649, MSE=0.00451458
##
## Node number 6: 10451 observations, complexity param=0.02957519
## mean=0.223615, MSE=0.1736113
## left son=12 (9336 obs) right son=13 (1115 obs)
## Primary splits:
## Inflight.wifi.service < 4.5 to the left, improve=0.41465830, (0 missing)
## Ease.of.Online.booking < 4.5 to the left, improve=0.25583170, (0 missing)
## Age < 41.5 to the right, improve=0.04362807, (0 missing)
## Arrival.Delay.in.Minutes_log < 1.868835 to the right, improve=0.03946246, (0 missing)
## Leg.room.service < 3.5 to the left, improve=0.03694485, (0 missing)
## Surrogate splits:
## Ease.of.Online.booking < 4.5 to the left, agree=0.949, adj=0.518, (0 split)
##
## Node number 7: 40872 observations, complexity param=0.02820567
## mean=0.8508514, MSE=0.1269033
## left son=14 (9989 obs) right son=15 (30883 obs)
## Primary splits:
## Inflight.entertainment < 3.5 to the left, improve=0.1383364, (0 missing)
## On.board.service < 3.5 to the left, improve=0.1181314, (0 missing)
## Inflight.service < 3.5 to the left, improve=0.1163792, (0 missing)
## Baggage.handling < 3.5 to the left, improve=0.1120047, (0 missing)
## Seat.comfort < 3.5 to the left, improve=0.1105292, (0 missing)
## Surrogate splits:
## Inflight.service < 3.5 to the left, agree=0.821, adj=0.267, (0 split)
## Seat.comfort < 3.5 to the left, agree=0.820, adj=0.265, (0 split)
## Cleanliness < 2.5 to the left, agree=0.813, adj=0.233, (0 split)
## Baggage.handling < 3.5 to the left, agree=0.810, adj=0.223, (0 split)
## On.board.service < 3.5 to the left, agree=0.797, adj=0.170, (0 split)
##
## Node number 8: 45789 observations, complexity param=0.01490694
## mean=0.06490642, MSE=0.06069358
## left son=16 (32928 obs) right son=17 (12861 obs)
## Primary splits:
## Class splits as RLL, improve=0.09851169, (0 missing)
## Type.of.Travel splits as RL, improve=0.05430057, (0 missing)
## Flight.Distance_log < 7.376821 to the left, improve=0.05282616, (0 missing)
## Leg.room.service < 3.5 to the left, improve=0.03067756, (0 missing)
## Departure.Arrival.time.convenient < 3.5 to the right, improve=0.03004393, (0 missing)
## Surrogate splits:
## Flight.Distance_log < 7.470509 to the left, agree=0.778, adj=0.208, (0 split)
## Ease.of.Online.booking < 3.5 to the left, agree=0.723, adj=0.012, (0 split)
## Age < 78.5 to the left, agree=0.719, adj=0.001, (0 split)
## Arrival.Delay.in.Minutes_log < 6.547471 to the left, agree=0.719, adj=0.000, (0 split)
## Inflight.entertainment < 0.5 to the right, agree=0.719, adj=0.000, (0 split)
##
## Node number 9: 4718 observations
## mean=0.6477321, MSE=0.2281752
##
## Node number 12: 9336 observations
## mean=0.1308912, MSE=0.1137587
##
## Node number 13: 1115 observations
## mean=1, MSE=0
##
## Node number 14: 9989 observations, complexity param=0.01904935
## mean=0.6178797, MSE=0.2361044
## left son=28 (6957 obs) right son=29 (3032 obs)
## Primary splits:
## Online.boarding < 4.5 to the left, improve=0.20547210, (0 missing)
## Checkin.service < 2.5 to the left, improve=0.17629500, (0 missing)
## Inflight.wifi.service < 4.5 to the left, improve=0.12251280, (0 missing)
## Seat.comfort < 4.5 to the left, improve=0.11016760, (0 missing)
## Cleanliness < 2.5 to the left, improve=0.07822242, (0 missing)
## Surrogate splits:
## Inflight.wifi.service < 4.5 to the left, agree=0.779, adj=0.271, (0 split)
## Ease.of.Online.booking < 4.5 to the left, agree=0.736, adj=0.130, (0 split)
## Cleanliness < 4.5 to the left, agree=0.725, adj=0.095, (0 split)
## Seat.comfort < 4.5 to the left, agree=0.721, adj=0.082, (0 split)
## Checkin.service < 4.5 to the left, agree=0.699, adj=0.009, (0 split)
##
## Node number 15: 30883 observations
## mean=0.9262054, MSE=0.06834899
##
## Node number 16: 32928 observations
## mean=0.01658163, MSE=0.01630668
##
## Node number 17: 12861 observations, complexity param=0.01490694
## mean=0.1886323, MSE=0.1530502
## left son=34 (9015 obs) right son=35 (3846 obs)
## Primary splits:
## Inflight.entertainment < 3.5 to the left, improve=0.20442150, (0 missing)
## Leg.room.service < 3.5 to the left, improve=0.16640300, (0 missing)
## On.board.service < 3.5 to the left, improve=0.15337020, (0 missing)
## Inflight.service < 3.5 to the left, improve=0.09739190, (0 missing)
## Baggage.handling < 3.5 to the left, improve=0.09394271, (0 missing)
## Surrogate splits:
## Cleanliness < 3.5 to the left, agree=0.828, adj=0.426, (0 split)
## Food.and.drink < 3.5 to the left, agree=0.790, adj=0.299, (0 split)
## Seat.comfort < 3.5 to the left, agree=0.784, adj=0.279, (0 split)
## On.board.service < 3.5 to the left, agree=0.745, adj=0.148, (0 split)
## Inflight.service < 4.5 to the left, agree=0.739, adj=0.126, (0 split)
##
## Node number 28: 6957 observations, complexity param=0.01055337
## mean=0.4724738, MSE=0.2492423
## left son=56 (1852 obs) right son=57 (5105 obs)
## Primary splits:
## Checkin.service < 2.5 to the left, improve=0.15482680, (0 missing)
## Seat.comfort < 4.5 to the left, improve=0.13168950, (0 missing)
## Cleanliness < 2.5 to the left, improve=0.09278151, (0 missing)
## Inflight.wifi.service < 4.5 to the left, improve=0.07119093, (0 missing)
## Class splits as RLL, improve=0.06046703, (0 missing)
## Surrogate splits:
## Age < 68.5 to the right, agree=0.735, adj=0.003, (0 split)
## Leg.room.service < 0.5 to the left, agree=0.734, adj=0.001, (0 split)
##
## Node number 29: 3032 observations
## mean=0.9515172, MSE=0.04613226
##
## Node number 34: 9015 observations
## mean=0.07310039, MSE=0.06775672
##
## Node number 35: 3846 observations, complexity param=0.01490694
## mean=0.4594384, MSE=0.2483548
## left son=70 (1556 obs) right son=71 (2290 obs)
## Primary splits:
## Customer.Type splits as LR, improve=0.4831552, (0 missing)
## Leg.room.service < 3.5 to the left, improve=0.3100923, (0 missing)
## Age < 32.5 to the left, improve=0.2467983, (0 missing)
## Flight.Distance_log < 7.345042 to the left, improve=0.2363762, (0 missing)
## Food.and.drink < 3.5 to the right, improve=0.2089187, (0 missing)
## Surrogate splits:
## Food.and.drink < 3.5 to the right, agree=0.727, adj=0.325, (0 split)
## On.board.service < 3.5 to the left, agree=0.713, adj=0.292, (0 split)
## Cleanliness < 3.5 to the right, agree=0.712, adj=0.289, (0 split)
## Leg.room.service < 3.5 to the left, agree=0.706, adj=0.274, (0 split)
## Flight.Distance_log < 7.24136 to the left, agree=0.706, adj=0.273, (0 split)
##
## Node number 56: 1852 observations
## mean=0.1463283, MSE=0.1249163
##
## Node number 57: 5105 observations
## mean=0.5907933, MSE=0.2417566
##
## Node number 70: 1556 observations
## mean=0.03920308, MSE=0.0376662
##
## Node number 71: 2290 observations
## mean=0.7449782, MSE=0.1899857print(tree_model_all$variable.importance)## Online.boarding Inflight.wifi.service
## 9.076455e+03 7.261304e+03
## Seat.comfort Inflight.entertainment
## 4.223983e+03 3.780147e+03
## Ease.of.Online.booking Class
## 3.608243e+03 3.545249e+03
## Type.of.Travel Cleanliness
## 3.274419e+03 5.219502e+02
## Customer.Type On.board.service
## 4.614965e+02 3.160414e+02
## Age Checkin.service
## 2.789687e+02 2.727817e+02
## Food.and.drink Inflight.service
## 2.700948e+02 2.422767e+02
## Leg.room.service Flight.Distance_log
## 1.998079e+02 1.830583e+02
## Baggage.handling Arrival.Delay.in.Minutes_log
## 1.597525e+02 8.514848e-02중요도를 순서대로 5개의 변수를 확인했을 때, [“Online.boarding”, “Inflight.wifi.service”, “Seat.comfort”, “Inflight.entertainment”, “Ease.of.Online.booking”]임을 알 수 있었다.
위의 5가지 변수들만을 포함하여, 새로운 tree model을 만들기로 한다.
tree_model_selected = rpart(satisfaction ~ Online.boarding + Inflight.wifi.service + Seat.comfort + Inflight.entertainment + Ease.of.Online.booking, data = airline_train_set, method = "class")summary(tree_model_selected)## Call:
## rpart(formula = satisfaction ~ Online.boarding + Inflight.wifi.service +
## Seat.comfort + Inflight.entertainment + Ease.of.Online.booking,
## data = airline_train_set, method = "class")
## n= 103594
##
## CP nsplit rel error xerror xstd
## 1 0.51012317 0 1.0000000 1.0000000 0.003552483
## 2 0.03893356 1 0.4898768 0.4898768 0.002931653
## 3 0.03705147 2 0.4509433 0.4509433 0.002842710
## 4 0.03104885 4 0.3768403 0.3768403 0.002650023
## 5 0.01599216 5 0.3457915 0.3457915 0.002558838
## 6 0.01000000 6 0.3297993 0.3297993 0.002509133
##
## Variable importance
## Online.boarding Inflight.wifi.service Seat.comfort
## 30 26 15
## Inflight.entertainment Ease.of.Online.booking
## 15 13
##
## Node number 1: 103594 observations, complexity param=0.5101232
## predicted class=0 expected loss=0.4333938 P(node) =1
## class counts: 58697 44897
## probabilities: 0.567 0.433
## left son=2 (52271 obs) right son=3 (51323 obs)
## Primary splits:
## Online.boarding < 3.5 to the left, improve=17076.920, (0 missing)
## Inflight.entertainment < 3.5 to the left, improve= 8782.078, (0 missing)
## Inflight.wifi.service < 3.5 to the left, improve= 8531.767, (0 missing)
## Seat.comfort < 3.5 to the left, improve= 7471.493, (0 missing)
## Ease.of.Online.booking < 3.5 to the left, improve= 3305.320, (0 missing)
## Surrogate splits:
## Seat.comfort < 3.5 to the left, agree=0.730, adj=0.455, (0 split)
## Inflight.wifi.service < 3.5 to the left, agree=0.714, adj=0.424, (0 split)
## Ease.of.Online.booking < 3.5 to the left, agree=0.676, adj=0.346, (0 split)
## Inflight.entertainment < 3.5 to the left, agree=0.659, adj=0.311, (0 split)
##
## Node number 2: 52271 observations, complexity param=0.03893356
## predicted class=0 expected loss=0.1489162 P(node) =0.5045756
## class counts: 44487 7784
## probabilities: 0.851 0.149
## left son=4 (50507 obs) right son=5 (1764 obs)
## Primary splits:
## Inflight.wifi.service < 0.5 to the right, improve=2616.62600, (0 missing)
## Ease.of.Online.booking < 0.5 to the right, improve=1119.89000, (0 missing)
## Online.boarding < 0.5 to the right, improve= 843.53630, (0 missing)
## Inflight.entertainment < 3.5 to the left, improve= 543.38630, (0 missing)
## Seat.comfort < 4.5 to the left, improve= 26.70475, (0 missing)
## Surrogate splits:
## Ease.of.Online.booking < 0.5 to the right, agree=0.971, adj=0.151, (0 split)
## Online.boarding < 0.5 to the right, agree=0.968, adj=0.041, (0 split)
## Inflight.entertainment < 0.5 to the right, agree=0.966, adj=0.002, (0 split)
##
## Node number 3: 51323 observations, complexity param=0.03705147
## predicted class=1 expected loss=0.2768739 P(node) =0.4954244
## class counts: 14210 37113
## probabilities: 0.277 0.723
## left son=6 (16097 obs) right son=7 (35226 obs)
## Primary splits:
## Inflight.entertainment < 3.5 to the left, improve=3047.6900, (0 missing)
## Inflight.wifi.service < 4.5 to the left, improve=1787.6230, (0 missing)
## Seat.comfort < 3.5 to the left, improve=1535.5770, (0 missing)
## Online.boarding < 4.5 to the left, improve=1527.3160, (0 missing)
## Ease.of.Online.booking < 4.5 to the left, improve= 580.3295, (0 missing)
## Surrogate splits:
## Seat.comfort < 3.5 to the left, agree=0.786, adj=0.318, (0 split)
## Ease.of.Online.booking < 0.5 to the left, agree=0.689, adj=0.007, (0 split)
## Inflight.wifi.service < 0.5 to the left, agree=0.688, adj=0.006, (0 split)
##
## Node number 4: 50507 observations, complexity param=0.03104885
## predicted class=0 expected loss=0.1193498 P(node) =0.4875475
## class counts: 44479 6028
## probabilities: 0.881 0.119
## left son=8 (45789 obs) right son=9 (4718 obs)
## Primary splits:
## Inflight.wifi.service < 3.5 to the left, improve=2905.861000, (0 missing)
## Ease.of.Online.booking < 3.5 to the left, improve= 800.324200, (0 missing)
## Inflight.entertainment < 3.5 to the left, improve= 544.586800, (0 missing)
## Seat.comfort < 4.5 to the left, improve= 18.642340, (0 missing)
## Online.boarding < 2.5 to the left, improve= 6.842733, (0 missing)
##
## Node number 5: 1764 observations
## predicted class=1 expected loss=0.004535147 P(node) =0.01702801
## class counts: 8 1756
## probabilities: 0.005 0.995
##
## Node number 6: 16097 observations, complexity param=0.03705147
## predicted class=0 expected loss=0.4682239 P(node) =0.1553854
## class counts: 8560 7537
## probabilities: 0.532 0.468
## left son=12 (13791 obs) right son=13 (2306 obs)
## Primary splits:
## Inflight.wifi.service < 4.5 to the left, improve=1519.8040, (0 missing)
## Online.boarding < 4.5 to the left, improve= 794.8197, (0 missing)
## Ease.of.Online.booking < 4.5 to the left, improve= 614.7695, (0 missing)
## Seat.comfort < 4.5 to the left, improve= 180.4196, (0 missing)
## Inflight.entertainment < 2.5 to the left, improve= 139.9171, (0 missing)
## Surrogate splits:
## Ease.of.Online.booking < 4.5 to the left, agree=0.919, adj=0.436, (0 split)
##
## Node number 7: 35226 observations
## predicted class=1 expected loss=0.1603929 P(node) =0.340039
## class counts: 5650 29576
## probabilities: 0.160 0.840
##
## Node number 8: 45789 observations
## predicted class=0 expected loss=0.06490642 P(node) =0.4420044
## class counts: 42817 2972
## probabilities: 0.935 0.065
##
## Node number 9: 4718 observations
## predicted class=1 expected loss=0.3522679 P(node) =0.04554318
## class counts: 1662 3056
## probabilities: 0.352 0.648
##
## Node number 12: 13791 observations, complexity param=0.01599216
## predicted class=0 expected loss=0.3793779 P(node) =0.1331255
## class counts: 8559 5232
## probabilities: 0.621 0.379
## left son=24 (13073 obs) right son=25 (718 obs)
## Primary splits:
## Inflight.wifi.service < 0.5 to the right, improve=583.4847, (0 missing)
## Ease.of.Online.booking < 0.5 to the right, improve=389.3154, (0 missing)
## Seat.comfort < 4.5 to the left, improve=232.0426, (0 missing)
## Online.boarding < 4.5 to the left, improve=215.4442, (0 missing)
## Inflight.entertainment < 2.5 to the left, improve=160.5671, (0 missing)
## Surrogate splits:
## Ease.of.Online.booking < 0.5 to the right, agree=0.988, adj=0.773, (0 split)
##
## Node number 13: 2306 observations
## predicted class=1 expected loss=0.0004336513 P(node) =0.02225998
## class counts: 1 2305
## probabilities: 0.000 1.000
##
## Node number 24: 13073 observations
## predicted class=0 expected loss=0.3452918 P(node) =0.1261946
## class counts: 8559 4514
## probabilities: 0.655 0.345
##
## Node number 25: 718 observations
## predicted class=1 expected loss=0 P(node) =0.006930903
## class counts: 0 718
## probabilities: 0.000 1.000train_pred_all = predict(tree_model_selected, airline_train_set, type = "class")
test_pred_all = predict(tree_model_selected, airline_test_set, type = "class")
train_conf_matrix_all = table(airline_train_set$satisfaction, train_pred_all)
test_conf_matrix_all = table(airline_test_set$satisfaction, test_pred_all)
# 훈련 오류율 계산
train_error_rate = 1 - sum(diag(train_conf_matrix_all)) / sum(train_conf_matrix_all)
test_error_rate = 1 - sum(diag(test_conf_matrix_all)) / sum(test_conf_matrix_all)
cat("train Error Rate:", train_error_rate, "\n")## train Error Rate: 0.142933table(train_pred_all)## train_pred_all
## 0 1
## 58862 44732cat("test Error Rate:", test_error_rate, "\n")## test Error Rate: 0.1467578# 단말 노드의 개수 확인
num_terminal_nodes = sum(tree_model_selected$frame$var == "<leaf>")
cat("Number of Terminal Nodes:", num_terminal_nodes, "\n")## Number of Terminal Nodes: 7변수를 선택해서 넣은 경우의 training error rate: 14%, test error
rate: 15%
상당히 낮은 error rate을 보인다.
매우 잘 동작하는 것을 확인할 수 있다.
rpart.plot(tree_model_selected)
print(tree_model_selected$variable.importance)## Online.boarding Inflight.wifi.service Seat.comfort
## 17183.720 14878.110 8731.368
## Inflight.entertainment Ease.of.Online.booking
## 8368.055 7440.170root node : Online.boarding(기준 : 4)
온라인 탑승 수속 만족도가 4보다 작다면 왼쪽, 크다면 오른쪽으로 데이터를
분할한다.
satisfaction의 예측에 있어서 해당 모델에서 Online.boarding이 가장 중요한 변수로 사용되었다. 만족도에 있어서 Online.boarding(온라인 탑승 수속 만족도)이 중요한 요소라고 할 수 있다.
#tree_model_selected에 대한 confusion matrix
test_conf_matrix_all## test_pred_all
## 0 1
## 0 12671 1857
## 1 1943 9422TP = 9422
TN = 12671
FP = 1857
FN = 1943
accuracy = (TP + TN) / (TP + TN + FP + FN)
precision = TP / (TP + FP)
recall = TP / (TP + FN)
specificity = TN / (TN + FP)
f1_score = 2 * (precision * recall) / (precision + recall)
print(paste("Accuracy:", accuracy))## [1] "Accuracy: 0.853242189008612"print(paste("Precision:", precision))## [1] "Precision: 0.835357744480894"print(paste("Recall:", recall))## [1] "Recall: 0.829036515618126"print(paste("Specificity:", specificity))## [1] "Specificity: 0.872177863436123"print(paste("F1 Score:", f1_score))## [1] "F1 Score: 0.832185126302773"satisfaction을 예측하는 tree model의 경우, f1 score가 0.83, accuracy가 0.85로 이전의 satisfaction을 예측하는 모델 중에서 가장 좋은 성능을 보이는 것을 확인할 수 있다.